Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BSE-4358: BodoSQL support for S3 tables #126

Merged
merged 81 commits into from
Jan 10, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
e76ddfb
Start planner API movement
njriasan Jan 2, 2025
64812e8
Updated the entry point for parseQuery
njriasan Jan 2, 2025
b1bbc8a
Fixed the import
njriasan Jan 2, 2025
88e6673
Merge branch 'main' into nick/bodosql_python_interface
njriasan Jan 2, 2025
325486c
Merge branch 'main' into nick/bodosql_python_interface
njriasan Jan 2, 2025
85cbbad
Moved getOptimizedPlanString
njriasan Jan 3, 2025
777458d
Moved getPandasAndPlanString
njriasan Jan 3, 2025
293e3b8
Moved getPandasString
njriasan Jan 3, 2025
2a835ec
Moved getWriteType
njriasan Jan 3, 2025
f9d9611
Moved executeDDL
njriasan Jan 3, 2025
abab715
Moved the last RelationalAlgebra API
njriasan Jan 3, 2025
1ec8501
Added missing docstrings [run CI]
njriasan Jan 3, 2025
7f9d2e9
Fixed the import ordering [run CI]
njriasan Jan 3, 2025
7da882e
Move reset to package private [run CI]
njriasan Jan 3, 2025
6d02755
Updated fromTypeID API [run CI]
njriasan Jan 3, 2025
2a943b9
Updated WriteTargetEnum [run CI]
njriasan Jan 3, 2025
add7295
Updated the logger API [run CI]
njriasan Jan 3, 2025
d8665d5
Fixed save issue [run CI]
njriasan Jan 3, 2025
28cce7d
Fixed API copying issue [run CI]
njriasan Jan 3, 2025
7ac23dc
Merge branch 'nick/bodosql_python_interface' into nick/fix_BodoSQLCol…
njriasan Jan 3, 2025
af1a9cf
Added missing static declaration [run CI]
njriasan Jan 3, 2025
864ee4e
Merge branch 'nick/bodosql_python_interface' into nick/fix_BodoSQLCol…
njriasan Jan 3, 2025
48605d6
Added more missing static declarations [run CI]
njriasan Jan 3, 2025
1b909c9
Merge branch 'main' into nick/bodosql_python_interface [run CI]
njriasan Jan 3, 2025
04db869
Removed ArrayList
njriasan Jan 3, 2025
80fd441
Removed hash map interaction
njriasan Jan 3, 2025
98bafde
Added stack trace API
njriasan Jan 3, 2025
052ce93
Removed properties
njriasan Jan 3, 2025
c25928c
Added builders
njriasan Jan 3, 2025
2349f14
Add local table builder
njriasan Jan 3, 2025
142c45f
Update LocalSchema
njriasan Jan 3, 2025
7febfe5
Removed LocalSchemaClass [run CI]
njriasan Jan 3, 2025
f87b383
Removed more dead code [run CI]
njriasan Jan 3, 2025
bca4fdc
Fixed bugs [run CI]
njriasan Jan 3, 2025
3b36b27
Merge branch 'nick/bodosql_python_interface' into nick/fix_BodoSQLCol…
njriasan Jan 3, 2025
688f131
Merge branch 'nick/fix_BodoSQLColumn' into nick/remove_constructors_b…
njriasan Jan 3, 2025
6918eb9
Removed planner type
njriasan Jan 3, 2025
09f51b6
Removed DDLExecutionResult calls
njriasan Jan 3, 2025
7ebec08
Added get lowered globals
njriasan Jan 3, 2025
f484159
Refactored pair access
njriasan Jan 3, 2025
1c5d5be
Fixed the exception interface
njriasan Jan 3, 2025
b1cee44
Removed ColumnDataTypeClass
njriasan Jan 3, 2025
76a0c56
Removed ColumnDataTypeClass
njriasan Jan 3, 2025
1e0139e
Removed everything but the constructors
njriasan Jan 3, 2025
f63733d
Refactored RelationalAlgebraGenerator constructor
njriasan Jan 3, 2025
374e4bf
Removed RelationalAlgebraGeneratorClass [run CI]
njriasan Jan 3, 2025
db61339
Merge branch 'main' into nick/bodosql_python_interface [run CI]
njriasan Jan 4, 2025
4b7890e
Merge branch 'nick/bodosql_python_interface' into nick/fix_BodoSQLCol…
njriasan Jan 4, 2025
a9c46f3
Merge branch 'nick/fix_BodoSQLColumn' into nick/remove_constructors_b…
njriasan Jan 4, 2025
10952b6
Merge branch 'nick/remove_constructors_bodosql' into nick/remove_rema…
njriasan Jan 4, 2025
395d644
Fixed map APIs [run CI]
njriasan Jan 5, 2025
95052cb
Merge branch 'nick/remove_constructors_bodosql' into nick/remove_rema…
njriasan Jan 5, 2025
2ca5b36
Fixed create_java_dynamic_parameter_type_list [run CI]
njriasan Jan 6, 2025
db9edfa
Merge branch 'nick/remove_constructors_bodosql' into nick/remove_rema…
njriasan Jan 6, 2025
d066870
Fixed merge conflict [run CI]
njriasan Jan 6, 2025
c873e60
Merge branch 'nick/fix_BodoSQLColumn' into nick/remove_constructors_b…
njriasan Jan 6, 2025
2dfb822
Apply Isaac's feedback [run CI]
njriasan Jan 6, 2025
c66bafc
Merge branch 'nick/remove_constructors_bodosql' into nick/remove_rema…
njriasan Jan 6, 2025
fcebae7
Merged with main [run CI]
njriasan Jan 6, 2025
0fb2a07
Merge branch 'nick/remove_constructors_bodosql' into nick/remove_rema…
njriasan Jan 6, 2025
2fddacb
Fixed APIs
njriasan Jan 6, 2025
3164ada
moved definitions [run CI]
njriasan Jan 6, 2025
86947e4
Add S3TablesCatalog skeleton
IsaacWarren Jan 6, 2025
e91d009
Add implementations for bodosql catalog
IsaacWarren Jan 6, 2025
0f7b77c
Add BodoS3TablesCatalog to PythonEntryPoint
IsaacWarren Jan 7, 2025
550691b
Add s3_tables_catalog to bodosql
IsaacWarren Jan 7, 2025
0585d51
Add s3_tables_catalog fixture
IsaacWarren Jan 7, 2025
01fe6ac
Actually cleanup written table
IsaacWarren Jan 7, 2025
d712c80
Add basic s3 tables read test
IsaacWarren Jan 7, 2025
a7cd361
Add write test
IsaacWarren Jan 7, 2025
43844d7
Update docs
IsaacWarren Jan 7, 2025
e9372c4
Allow catalogs to not specify a default schema
IsaacWarren Jan 7, 2025
d5bb594
Don't give default schema
IsaacWarren Jan 7, 2025
c015ad2
[Run CI]
IsaacWarren Jan 7, 2025
43f60fc
Merge remote-tracking branch 'origin/main' into isaac/bodosql_s3_tables
IsaacWarren Jan 8, 2025
5a8c201
[Run CI]
IsaacWarren Jan 8, 2025
6e2cc69
Fix docstring
IsaacWarren Jan 9, 2025
21de7a1
Fix doc
IsaacWarren Jan 9, 2025
d4389f5
Fix comment
IsaacWarren Jan 9, 2025
06c9ed3
Add infra requirements to docstrings
IsaacWarren Jan 9, 2025
7095e2d
[Run CI]
IsaacWarren Jan 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Moved executeDDL
  • Loading branch information
njriasan committed Jan 3, 2025
commit f9d9611fa46295f008c120570d04e20615483ccc
2 changes: 1 addition & 1 deletion BodoSQL/bodosql/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -1416,7 +1416,7 @@ def execute_ddl(

if bodo.get_rank() == 0:
try:
ddl_result = generator.executeDDL(sql)
ddl_result = JavaEntryPoint.executeDDL(generator, sql)
# Convert the output to a DataFrame.
column_names = list(ddl_result.getColumnNames())
column_types = [
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package com.bodosql.calcite.application

import com.bodosql.calcite.ddl.DDLExecutionResult
import com.bodosql.calcite.table.ColumnDataTypeInfo

/**
Expand Down Expand Up @@ -113,5 +114,19 @@ class PythonEntryPoint {
generator: RelationalAlgebraGenerator,
sql: String,
): String = generator.getWriteType(sql)

/**
* Execute the given DDL statement in an interpreted manner. This
* assumes/requires that sql is a DDL statement, which should have
* already been checked.
* @param generator The generator to use.
* @param sql The DDL statement to execute
* @return The result of the DDL execution.
*/
@JvmStatic
fun executeDDL(
generator: RelationalAlgebraGenerator,
sql: String,
): DDLExecutionResult = generator.executeDDL(sql)
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -685,6 +685,15 @@ PandasCodeSqlPlanPair getPandasAndPlanString(
}
}

/**
* Return the Python code generated for the given SQL query.
*
* @param sql The SQL query to process.
* @param dynamicParamTypes The dynamic parameter types.
* @param namedParamTypeMap The named parameter types.
* @return The Python code generated for the given SQL query.
* @throws Exception If an error occurs while processing the SQL query.
*/
String getPandasString(
@NonNull String sql,
@NonNull List<ColumnDataTypeInfo> dynamicParamTypes,
Expand Down Expand Up @@ -732,9 +741,14 @@ String getWriteType(String sql) throws Exception {
}
}

// ~~~~~~~~~~~~~PYTHON EXPOSED APIS~~~~~~~~~~~~~~

public DDLExecutionResult executeDDL(String sql) throws Exception {
/**
* Execute the given DDL statement in an interpreted manner. This assumes/requires that sql is a
* DDL statement, which should have already been checked.
*
* @param sql The DDL statement to execute
* @return The result of the DDL execution.
*/
DDLExecutionResult executeDDL(String sql) throws Exception {
try {
// DDL doesn't support dynamic or named parameters at this time.
SqlNode validatedSqlNode = validateQuery(sql, List.of(), Map.of());
Expand All @@ -751,6 +765,8 @@ public DDLExecutionResult executeDDL(String sql) throws Exception {
}
}

// ~~~~~~~~~~~~~PYTHON EXPOSED APIS~~~~~~~~~~~~~~

/**
* Determine if the active query is a DDL query that is not treated like compute (not CTAS).
*
Expand Down