feat: support bulk-export command#261
Open
ohadedry wants to merge 10 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new bulk-export command to Fabric CLI (fab) to export all supported items from a workspace or folder in one operation, preserving folder structure and item bindings. It includes end-to-end command wiring (parser → command handler → API client), export utilities for writing bulk-export definition parts to storage, debug-log redaction improvements, and corresponding documentation + tests (including VCR recordings).
Changes:
- Implemented
bulk-exportcommand flow (parsing, command dispatch, workspace/folder execution paths, API call, and export-to-storage utilities). - Extended export utilities and HTTP debug logging to better support bulk export payload shapes and reduce noisy/large debug logs.
- Added docs pages/examples and comprehensive tests + recordings for the new command.
Reviewed changes
Copilot reviewed 39 out of 39 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_utils/test_fab_cmd_bulk_export_utils.py | Unit tests for bulk-export utility helpers (payload creation, summary output, path stripping/export). |
| tests/test_core/test_fab_logger.py | Updates logger tests for new command-context-aware response logging/redaction. |
| tests/test_commands/test_bulk_export.py | New command-level tests for bulk-export behavior and output structure. |
| tests/test_commands/recordings/test_commands/test_bulk_export/test_bulk_export_workspace_without_recursive_fail.yaml | VCR recording for workspace bulk-export missing --recursive. |
| tests/test_commands/recordings/test_commands/test_bulk_export/test_bulk_export_workspace_folder_without_recursive_fail.yaml | VCR recording for folder bulk-export missing --recursive. |
| tests/test_commands/recordings/test_commands/test_bulk_export/test_bulk_export_output_path_not_empty_warning.yaml | VCR recording for warning behavior when output directory is non-empty. |
| tests/test_commands/recordings/test_commands/test_bulk_export/test_bulk_export_no_exportable_items_fail.yaml | VCR recording for “no exportable items” failure case. |
| tests/test_commands/recordings/test_commands/test_bulk_export/test_bulk_export_item_fail.yaml | VCR recording for invalid target type (item path). |
| tests/test_commands/recordings/test_commands/test_bulk_export/test_bulk_export_empty_workspace_fail.yaml | VCR recording for empty workspace failure case. |
| tests/test_commands/recordings/test_commands/test_bulk_export/test_bulk_export_empty_folder_fail.yaml | VCR recording for empty folder failure case. |
| tests/test_commands/recordings/test_commands/test_bulk_export/class_setup.yaml | VCR class setup recording for bulk-export command tests. |
| tests/test_commands/commands_parser.py | Registers the new bulk-export parser into the command test harness. |
| src/fabric_cli/utils/fab_commands.py | Adds bulk-export to the CLI command index text. |
| src/fabric_cli/utils/fab_cmd_export_utils.py | Extends decode/export helpers to support definitionParts payload shape. |
| src/fabric_cli/utils/fab_cmd_bulk_export_utils.py | New bulk-export utility module (payload, export writing, summary printing, path validation). |
| src/fabric_cli/parsers/fab_fs_parser.py | Adds argparse registration for the bulk-export command. |
| src/fabric_cli/errors/bulk_export.py | Defines bulk-export-specific error messages. |
| src/fabric_cli/core/fab_parser_setup.py | Wires bulk-export parser into the global CLI parser setup. |
| src/fabric_cli/core/fab_logger.py | Adds command context to response logging + JSON response redaction logic. |
| src/fabric_cli/core/fab_constant.py | Adds COMMAND_FS_BULKEXPORT_DESCRIPTION. |
| src/fabric_cli/core/fab_config/command_support.yaml | Registers bulk-export support matrix (workspace/folder + supported item types). |
| src/fabric_cli/core/fab_commands.py | Adds FS_BULKEXPORT command enum value. |
| src/fabric_cli/commands/fs/fab_fs.py | Hooks bulk_export_command into the fs command module. |
| src/fabric_cli/commands/fs/fab_fs_bulk_export.py | New bulk-export command implementation and precondition checks. |
| src/fabric_cli/commands/fs/bulk_export/fab_fs_bulk_export_workspace.py | Workspace bulk-export execution and response filtering. |
| src/fabric_cli/commands/fs/bulk_export/fab_fs_bulk_export_folder.py | Folder bulk-export execution and recursive item collection. |
| src/fabric_cli/commands/fs/bulk_export/init.py | Package init for bulk-export submodule. |
| src/fabric_cli/client/fab_api_item.py | Adds bulk_export_definitions() API wrapper. |
| src/fabric_cli/client/fab_api_client.py | Passes command context into HTTP response debug logging. |
| mkdocs.yml | Adds bulk-export page to MkDocs navigation. |
| docs/examples/workspace_examples.md | Adds workspace bulk-export example section. |
| docs/examples/folder_examples.md | Adds folder bulk-export example section. |
| docs/commands/index.md | Adds bulk-export to the command index page. |
| docs/commands/fs/bulk_export.md | New bulk-export command documentation page. |
ayeshurun
reviewed
Jul 6, 2026
ayeshurun
reviewed
Jul 8, 2026
ayeshurun
reviewed
Jul 8, 2026
ayeshurun
reviewed
Jul 8, 2026
Comment on lines
+55
to
+59
| if not item_definitions: | ||
| raise FabricCLIError( | ||
| BulkExportErrors.no_definition_returned(artifact_name), | ||
| fab_constant.ERROR_INVALID_DEFINITION_PAYLOAD, | ||
| ) |
Contributor
Author
There was a problem hiding this comment.
To me it's seems like an API error but I thought it will be good to indicate about it to the user, WDYT?
Contributor
Author
There was a problem hiding this comment.
To me it's seems like an API error but I thought it will be good to indicate about it to the user, WDYT?
Comment on lines
+60
to
+74
| if export_path["type"] == "local" and os.path.isdir(export_path["path"]): | ||
| is_export_path_empty = True | ||
| with os.scandir(export_path["path"]) as entries: | ||
| is_export_path_empty = not any(entries) | ||
| if not is_export_path_empty: | ||
| if not args.force: | ||
| if not fab_ui.prompt_confirm( | ||
| f"Output folder '{export_path['path']}' is not empty. Do you want to proceed?" | ||
| ): | ||
| return False | ||
| else: | ||
| export_path_warning = True | ||
| elif export_path["type"] == "local": | ||
| os.makedirs(export_path["path"], exist_ok=True) | ||
| else: |
Comment on lines
+74
to
+78
| "skipped": 1, | ||
| "skipped_types": { | ||
| "Dashboard": 1, | ||
| }, | ||
| "output": "/tmp" |
Comment on lines
+150
to
+151
| mock_print_warning.assert_called_with( | ||
| "Exporting to a non-empty output folder") |
Comment on lines
+219
to
+228
| export_path1 = ( | ||
| tmp_path / folder1.display_name / | ||
| f"{notebook1.display_name}.Notebook" | ||
| ) | ||
| export_path2 = ( | ||
| tmp_path | ||
| / folder1.display_name | ||
| / folder2.display_name | ||
| / f"{notebook2.display_name}.Notebook" | ||
| ) |
Comment on lines
+335
to
+336
| data = call_kwargs.kwargs.get( | ||
| "data", call_kwargs[1].get("data", [])) |
Comment on lines
+341
to
+344
| export_path = ( | ||
| tmp_path / folder.display_name / | ||
| f"{notebook.display_name}.Notebook" | ||
| ) |
Comment on lines
+184
to
+187
| export_path = ( | ||
| tmp_path / folder2.display_name / | ||
| f"{notebook2.display_name}.Notebook" | ||
| ) |
Comment on lines
+72
to
+73
| elif export_path["type"] == "local": | ||
| os.makedirs(export_path["path"], exist_ok=True) |
Comment on lines
+141
to
+151
| prefix = "/" + "/".join(segments) | ||
|
|
||
| for part in item_def.get("definitionParts", []): | ||
| path = part.get("path", "") | ||
| if path.startswith(prefix): | ||
| part["path"] = path[len(prefix) :] | ||
| else: | ||
| raise FabricCLIError( | ||
| BulkExportErrors.path_mismatch(), | ||
| fab_constant.ERROR_INVALID_DEFINITION_PAYLOAD, | ||
| ) |
| f"{item_type} ({count})" | ||
| for item_type, count in unsupported_types_count.items() | ||
| ] | ||
| output_format_message += f". Skipped {unsupported_count} items due to unsupported item types: {', '.join(unsupported_types)}" |
Comment on lines
+87
to
+96
| # find unsupported items that were not in the response index and add them to the unsupported_items_list | ||
| supported_item_ids: set[str] = {item.id for item in supported_items_list} | ||
| unsupported_item_ids: set[str] = {item.id for item in unsupported_items_list} | ||
| for ws_item in ws_items: | ||
| if ( | ||
| ws_item.id not in supported_item_ids | ||
| and ws_item.id not in unsupported_item_ids | ||
| ): | ||
| unsupported_items_list.append(ws_item) | ||
|
|
Comment on lines
130
to
+134
| monkeypatch.setattr(fab_state_config, "get_config", lambda x: "1") | ||
| logger.log_debug_http_response(200, {}, "Response text", time.time()) | ||
| logger.log_debug_http_response(200, {}, "Response text", time.time(), "cmd") | ||
|
|
||
| monkeypatch.setattr(fab_state_config, "get_config", lambda x: "0") | ||
| logger.log_debug_http_response(200, {}, "Response text", time.time()) | ||
| logger.log_debug_http_response(200, {}, "Response text", time.time(), "cmd") |
Comment on lines
+75
to
+78
| "skipped_types": { | ||
| "Dashboard": 1, | ||
| }, | ||
| "output": "/tmp" |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 39 out of 39 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
src/fabric_cli/utils/fab_cmd_export_utils.py:63
- The docstring still says this function exports from the
'parts'array, but the implementation now supports an arbitrary key viadefinition_parts(e.g.,definitionPartsfor bulk-export). Updating the docstring avoids misleading future callers.
"""
Export each 'payload' in the 'parts' array to a file named based on 'path' in 'parts'.
The 'payload' content will be saved as the file's content.
"""
Comment on lines
66
to
70
| register_rm_parser, | ||
| register_mkdir_parser, | ||
| register_jobs_parser, | ||
| register_bulk_export_parser, | ||
| ] |
Contributor
Author
There was a problem hiding this comment.
Where twice? Please explain
Comment on lines
+60
to
+78
| if export_path["type"] == "local" and os.path.isdir(export_path["path"]): | ||
| is_export_path_empty = True | ||
| with os.scandir(export_path["path"]) as entries: | ||
| is_export_path_empty = not any(entries) | ||
| if not is_export_path_empty: | ||
| if not args.force: | ||
| if not fab_ui.prompt_confirm( | ||
| f"Output folder '{export_path['path']}' is not empty. Do you want to proceed?" | ||
| ): | ||
| return False | ||
| else: | ||
| export_path_warning = True | ||
| elif export_path["type"] == "local": | ||
| os.makedirs(export_path["path"], exist_ok=True) | ||
| else: | ||
| raise FabricCLIError( | ||
| BulkExportErrors.invalid_export_path(export_path["path"]), | ||
| fab_constant.ERROR_INVALID_OPERATION, | ||
| ) |
Comment on lines
+87
to
+96
| # find unsupported items that were not in the response index and add them to the unsupported_items_list | ||
| supported_item_ids: set[str] = {item.id for item in supported_items_list} | ||
| unsupported_item_ids: set[str] = {item.id for item in unsupported_items_list} | ||
| for ws_item in ws_items: | ||
| if ( | ||
| ws_item.id not in supported_item_ids | ||
| and ws_item.id not in unsupported_item_ids | ||
| ): | ||
| unsupported_items_list.append(ws_item) | ||
|
|
Comment on lines
+56
to
+64
| export_path_warning = False | ||
| export_path = fab_storage.get_export_path(args.output) | ||
| if (export_path["type"] != "local") or ( | ||
| export_path["type"] == "local" and not os.path.isdir(export_path["path"]) | ||
| ): | ||
| raise FabricCLIError( | ||
| BulkExportErrors.invalid_export_path(export_path["path"]), | ||
| fab_constant.ERROR_INVALID_OPERATION, | ||
| ) |
Comment on lines
+89
to
+98
| # find unsupported items that were not in the response index and add them to the unsupported_items_list | ||
| supported_item_ids: set[str] = {item.id for item in supported_items_list} | ||
| unsupported_item_ids: set[str] = {item.id for item in unsupported_items_list} | ||
| for ws_item in ws_items: | ||
| if ( | ||
| ws_item.id not in supported_item_ids | ||
| and ws_item.id not in unsupported_item_ids | ||
| ): | ||
| unsupported_items_list.append(ws_item) | ||
|
|
Comment on lines
+57
to
61
| def export_json_parts( | ||
| args, definition: dict, export_path: dict, definition_parts="parts" | ||
| ) -> None: | ||
| """ | ||
| Export each 'payload' in the 'parts' array to a file named based on 'path' in 'parts'. |
Comment on lines
129
to
133
| def test_log_debug_http_response(monkeypatch): | ||
| monkeypatch.setattr(fab_state_config, "get_config", lambda x: "1") | ||
| logger.log_debug_http_response(200, {}, "Response text", time.time()) | ||
| logger.log_debug_http_response(200, {}, "Response text", time.time(), "cmd") | ||
|
|
||
| monkeypatch.setattr(fab_state_config, "get_config", lambda x: "0") |
Comment on lines
+89
to
+98
| # find unsupported items that were not in the response index and add them to the unsupported_items_list | ||
| supported_item_ids: set[str] = {item.id for item in supported_items_list} | ||
| unsupported_item_ids: set[str] = {item.id for item in unsupported_items_list} | ||
| for ws_item in ws_items: | ||
| if ( | ||
| ws_item.id not in supported_item_ids | ||
| and ws_item.id not in unsupported_item_ids | ||
| ): | ||
| unsupported_items_list.append(ws_item) | ||
|
|
Comment on lines
+61
to
+64
| ``` | ||
|
|
||
| When using `--output-format json`, the output includes structured data: | ||
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 39 out of 40 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
src/fabric_cli/utils/fab_cmd_export_utils.py:63
export_json_partsnow supports exporting from different list keys via thedefinition_partsparameter, but the docstring still describes only the'parts'array. Updating it to reflect the new parameter will prevent confusion for future callers and maintainers.
def export_json_parts(
args, definition: dict, export_path: dict, definition_parts="parts"
) -> None:
"""
Export each 'payload' in the 'parts' array to a file named based on 'path' in 'parts'.
The 'payload' content will be saved as the file's content.
"""
Comment on lines
+151
to
+159
| """Validate that all definition part paths concatenated with the export path are under the export path to prevent path traversal issues.""" | ||
| for part in item_def.get("definitionParts", []): | ||
| part_path = part.get("path", "").lstrip("/") | ||
| full_export_path = os.path.abspath(os.path.join(export_path, part_path)) | ||
| if not full_export_path.startswith(os.path.abspath(export_path) + os.sep): | ||
| raise FabricCLIError( | ||
| BulkExportErrors.path_mismatch_full_export_path(), | ||
| fab_constant.ERROR_INVALID_DEFINITION_PAYLOAD, | ||
| ) |
aviatco
reviewed
Jul 14, 2026
aviatco
reviewed
Jul 15, 2026
aviatco
reviewed
Jul 15, 2026
aviatco
reviewed
Jul 15, 2026
| f"{', '.join(unsupported_types)}" | ||
| ) | ||
| fab_ui.print_output_format( | ||
| args=args, |
Collaborator
There was a problem hiding this comment.
consider passing key-value format so it will be printed user friendly when format type is text
ohadedry
commented
Jul 15, 2026
Comment on lines
+58
to
+64
| if (export_path["type"] != "local") or ( | ||
| export_path["type"] == "local" and not os.path.isdir(export_path["path"]) | ||
| ): | ||
| raise FabricCLIError( | ||
| BulkExportErrors.invalid_export_path(export_path["path"]), | ||
| fab_constant.ERROR_INVALID_OPERATION, | ||
| ) |
Comment on lines
+48
to
+52
| # Recursively check for nested parts if applicable | ||
| if ( | ||
| "nested_parts" in part | ||
| ): # Assuming 'nested_parts' is a key for potential nested structures | ||
| decode_payload(part) |
Comment on lines
+153
to
+154
| full_export_path = os.path.abspath(os.path.join(export_path, part_path)) | ||
| if not full_export_path.startswith(os.path.abspath(export_path) + os.sep): |
Comment on lines
+89
to
+97
| # find unsupported items that were not in the response index and add them to the unsupported_items_list | ||
| supported_item_ids: set[str] = {item.id for item in supported_items_list} | ||
| unsupported_item_ids: set[str] = {item.id for item in unsupported_items_list} | ||
| for ws_item in ws_items: | ||
| if ( | ||
| ws_item.id not in supported_item_ids | ||
| and ws_item.id not in unsupported_item_ids | ||
| ): | ||
| unsupported_items_list.append(ws_item) |
ayeshurun
reviewed
Jul 16, 2026
Comment on lines
+355
to
+358
| # SQLDatabase creation modes | ||
| SQL_DATABASE_CREATION_MODE_NEW = "New" | ||
| SQL_DATABASE_CREATION_MODE_RESTORE = "Restore" | ||
| SQL_DATABASE_CREATION_MODE_RESTORE_DELETED = "RestoreDeletedDatabase" |
Collaborator
There was a problem hiding this comment.
what are these changes?
Comment on lines
+492
to
+494
| '$ deploy --config config.yml --target_env prod -P \'[{"param1":"value1"}]\' -f\n', | ||
| "# deploy using experimental bulk publish (single bulk import API call)", | ||
| "$ deploy --config config.yml --target_env dev --bulk_publish", |
Collaborator
There was a problem hiding this comment.
what are these changes? please rebase over main
Comment on lines
+55
to
+59
| if not item_definitions: | ||
| raise FabricCLIError( | ||
| BulkExportErrors.no_definition_returned(artifact_name), | ||
| fab_constant.ERROR_INVALID_DEFINITION_PAYLOAD, | ||
| ) |
ayeshurun
previously approved these changes
Jul 16, 2026
Comment on lines
+57
to
+78
| export_path = fab_storage.get_export_path(args.output) | ||
| if (export_path["type"] != "local") or ( | ||
| export_path["type"] == "local" and not os.path.isdir(export_path["path"]) | ||
| ): | ||
| raise FabricCLIError( | ||
| BulkExportErrors.invalid_export_path(export_path["path"]), | ||
| fab_constant.ERROR_INVALID_OPERATION, | ||
| ) | ||
| if os.path.isdir(export_path["path"]): | ||
| is_export_path_empty = True | ||
| with os.scandir(export_path["path"]) as entries: | ||
| is_export_path_empty = not any(entries) | ||
| if not is_export_path_empty: | ||
| if not args.force: | ||
| if not fab_ui.prompt_confirm( | ||
| f"Output folder '{export_path['path']}' is not empty. Do you want to proceed?" | ||
| ): | ||
| return False | ||
| else: | ||
| export_path_warning = True | ||
| else: | ||
| os.makedirs(export_path["path"], exist_ok=True) |
Comment on lines
+89
to
+98
| # find unsupported items that were not in the response index and add them to the unsupported_items_list | ||
| supported_item_ids: set[str] = {item.id for item in supported_items_list} | ||
| unsupported_item_ids: set[str] = {item.id for item in unsupported_items_list} | ||
| for ws_item in ws_items: | ||
| if ( | ||
| ws_item.id not in supported_item_ids | ||
| and ws_item.id not in unsupported_item_ids | ||
| ): | ||
| unsupported_items_list.append(ws_item) | ||
|
|
Comment on lines
+152
to
+159
| for part in item_def.get("definitionParts", []): | ||
| part_path = part.get("path", "").lstrip("/") | ||
| full_export_path = os.path.abspath(os.path.join(export_path, part_path)) | ||
| if not full_export_path.startswith(os.path.abspath(export_path) + os.sep): | ||
| raise FabricCLIError( | ||
| BulkExportErrors.path_mismatch_full_export_path(), | ||
| fab_constant.ERROR_INVALID_DEFINITION_PAYLOAD, | ||
| ) |
| from fabric_cli.core import fab_constant | ||
| from fabric_cli.core.fab_exceptions import FabricCLIError | ||
| from fabric_cli.core.hiearchy.fab_folder import Folder | ||
| from fabric_cli.core.hiearchy.fab_hiearchy import FabricElement, Item, Workspace |
Comment on lines
+9
to
+16
| from fabric_cli.core.fab_exceptions import FabricCLIError | ||
| from fabric_cli.core import fab_constant | ||
| from fabric_cli.client import fab_api_item as item_api | ||
| from fabric_cli.errors.bulk_export import BulkExportErrors | ||
| from fabric_cli.core.hiearchy.fab_hiearchy import Item | ||
| from fabric_cli.utils import fab_cmd_fs_utils as utils_fs | ||
| from fabric_cli.utils.fab_cmd_bulk_export_utils import ContextItemsSupportMap | ||
| from fabric_cli.utils import fab_cmd_bulk_export_utils as bulk_export_utils |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
✨ Description of new changes
This pull request introduces a new
bulk-exportcommand to the CLI, enabling bulk export of all supported items from a workspace or folder while preserving folder structure and item bindings. The implementation includes API integration, argument validation, error handling, and comprehensive documentation and examples. The changes also update command support configuration and the command index.New Functionality:
bulk-exportcommand that allows users to export all supported items from a workspace or folder in a single operation, preserving folder structure and item bindings. This includes both CLI logic and integration with the backend API. [1] [2] [3] [4] [5]Documentation Updates:
bulk_export.mddetailing usage, parameters, supported item types, output structure, and examples for thebulk-exportcommand.bulk-export. [1] [2]Command Support and Configuration:
bulkexportcommand in the CLI command enumeration and in the command support YAML, specifying supported elements (workspace, folder) and item types. [1] [2]Internal Integration and Validation:
bulk-exportcommand into the main command handler and ensured argument validation, context checking, and user confirmation logic are in place. [1] [2] [3]Minor Improvements:
These changes collectively introduce and document the new
bulk-exportfeature, making it easier for users to export entire workspaces or folders with structure and bindings preserved.# 📥 Pull Request