- release_tracks: [ALPHA, BETA, GA] help_text: brief: | Create a Dataproc Metastore service. description: | Create a new Dataproc Metastore service with the given name and configurations. If run asynchronously with `--async`, exits after printing one operation name that can be used to poll the status of the creation via: {top_command} metastore operations describe examples: | To create a Dataproc Metastore service with the name `my-metastore-service` in location `us-central` using a non default port 9090, run: $ {command} my-metastore-service --location=us-central1 --port=9090 To create a Dataproc Metastore service with the name `my-metastore-service` in location `us-central` using a non default network foo, run: $ {command} my-metastore-service --location=us-central1 --network=foo request: ALPHA: api_version: v1alpha modify_request_hooks: - googlecloudsdk.command_lib.metastore.services.util:UpdateScalingConfigForCreate - googlecloudsdk.command_lib.metastore.services.util:GenerateNetworkConfigFromSubnetList - googlecloudsdk.command_lib.metastore.services.util:GenerateAuxiliaryVersionsConfigFromList - googlecloudsdk.command_lib.metastore.validators:ValidateServiceMutexConfig - googlecloudsdk.command_lib.metastore.validators:ValidateScheduledBackupConfigs - googlecloudsdk.command_lib.metastore.validators:ValidateKmsKeys BETA: api_version: v1beta modify_request_hooks: - googlecloudsdk.command_lib.metastore.services.util:UpdateScalingConfigForCreate - googlecloudsdk.command_lib.metastore.services.util:GenerateNetworkConfigFromSubnetList - googlecloudsdk.command_lib.metastore.services.util:GenerateAuxiliaryVersionsConfigFromList - googlecloudsdk.command_lib.metastore.validators:ValidateServiceMutexConfig - googlecloudsdk.command_lib.metastore.validators:ValidateScheduledBackupConfigs - googlecloudsdk.command_lib.metastore.validators:ValidateKmsKeys GA: api_version: v1 modify_request_hooks: - googlecloudsdk.command_lib.metastore.services.util:UpdateScalingConfigForCreate - googlecloudsdk.command_lib.metastore.services.util:GenerateNetworkConfigFromSubnetList - googlecloudsdk.command_lib.metastore.services.util:GenerateAuxiliaryVersionsConfigFromList - googlecloudsdk.command_lib.metastore.validators:ValidateServiceMutexConfigForV1 collection: metastore.projects.locations.services arguments: resource: help_text: | Arguments and flags that specify the Dataproc Metastore service you want to create. spec: !REF googlecloudsdk.command_lib.metastore.resources:service params: - arg_name: port api_field: service.port default: 9083 processor: googlecloudsdk.command_lib.metastore.validators:ValidatePort help_text: | The TCP port on which the Metastore service will listen. If unspecified, the default port 9083 will be used. - group: help_text: | The instance size of the Dataproc Metastore instance. mutex: true params: - arg_name: tier api_field: service.tier help_text: | The tier of the service. choices: - arg_value: enterprise enum_value: ENTERPRISE help_text: | The enterprise tier provides multi-zone high availability, and sufficient scalability for enterprise-level Dataproc Metastore workloads. - arg_value: developer enum_value: DEVELOPER help_text: | The developer tier provides limited scalability and no fault tolerance. Good for low-cost proof-of-concept. fallback: googlecloudsdk.core.properties:VALUES.metastore.tier.Get - arg_name: scaling-factor api_field: service.scalingConfig.scalingFactor help_text: | The scaling factor of the service. Can be one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0]. processor: googlecloudsdk.command_lib.metastore.validators:ValidateScalingFactor - arg_name: instance-size api_field: service.scalingConfig.instanceSize help_text: | The instance size of the service. choices: - arg_value: extra-small enum_value: EXTRA_SMALL help_text: | Extra small instance size, maps to a scaling factor of 0.1. - arg_value: small enum_value: SMALL help_text: | Small instance size, maps to a scaling factor of 0.5. - arg_value: medium enum_value: MEDIUM help_text: | Medium instance size, maps to a scaling factor of 1.0. - arg_value: large enum_value: LARGE help_text: | Large instance size, maps to a scaling factor of 3.0. - arg_value: extra-large enum_value: EXTRA_LARGE help_text: | Extra large instance size, maps to a scaling factor of 6.0. - group: help_text: | The autoscaling configuration of the Dataproc Metastore instance. params: - arg_name: min-scaling-factor api_field: service.scalingConfig.autoscalingConfig.limitConfig.minScalingFactor help_text: | The minimum scaling factor allowed by the autoscaler. Can be one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0]. The default value is 0.1. processor: googlecloudsdk.command_lib.metastore.validators:ValidateMinScalingFactor - arg_name: max-scaling-factor api_field: service.scalingConfig.autoscalingConfig.limitConfig.maxScalingFactor help_text: | The maximum scaling factor allowed by the autoscaler. Can be one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0]. The default value is 6.0. processor: googlecloudsdk.command_lib.metastore.validators:ValidateMaxScalingFactor - arg_name: autoscaling-enabled api_field: service.scalingConfig.autoscalingConfig.autoscalingEnabled help_text: | A boolean flag to determine whether Dataproc Metastore autoscaling should be enabled, false if unspecified. The default minimum and maximum scaling factors are 0.1 and 6.0, respectively. The minimum and maximum scaling factors can be specified using --min-scaling-factor and --max-scaling-factor. - arg_name: hive-metastore-version api_field: service.hiveMetastoreConfig.version help_text: | The Hive metastore schema version. The supported versions of a location are listed via: {top_command} metastore locations describe If unspecified, the default version chosen by the server will be used. - arg_name: data-catalog-sync default: false api_field: service.metadataIntegration.dataCatalogConfig.enabled help_text: | A boolean flag to determine whether Dataproc Metastore metadata sync to Data Catalog should be enabled, false if unspecified. Mutually exclusive with flag `--encryption-kms-key`. - arg_name: deletion-protection type: bool api_field: service.deletionProtection help_text: | Flag that enables delete protection on Dataproc Metastore instance to prevent accidental deletions of the instance. Use --deletion-protection to enable. - arg_name: release-channel help_text: | The release channel of the service. choices: - arg_value: stable enum_value: STABLE help_text: | The `STABLE` release channel contains features that are considered stable and have been validated for production use. - arg_value: canary enum_value: CANARY help_text: | The `CANARY` release channel contains the newest features, which may be unstable and subject to unresolved issues with no known workarounds. Services using the `CANARY` release channel are not subject to any SLAs. default: stable api_field: service.releaseChannel - arg_name: database-type help_text: | The type of database the Dataproc Metastore service will store data in. choices: - arg_value: mysql enum_value: MYSQL help_text: | `MYSQL` database type is a Dataproc Metastore service backed by MySQL CloudSQL. - arg_value: spanner enum_value: SPANNER help_text: | `SPANNER` database type is a Dataproc Metastore service backed by Cloud Spanner. default: mysql api_field: service.databaseType - group: help_text: | Configuration properties specific to running Hive metastore software as the metastore service. mutex: true params: - arg_name: hive-metastore-configs api_field: service.hiveMetastoreConfig.configOverrides.additionalProperties metavar: KEY=VALUE help_text: | A mapping of Hive metastore configuration key-value pairs to apply to the Hive metastore. type: arg_dict: flatten: true spec: - api_field: key - api_field: value - arg_name: hive-metastore-configs-from-file api_field: service.hiveMetastoreConfig.configOverrides type: "googlecloudsdk.calliope.arg_parsers:FileContents:" processor: googlecloudsdk.command_lib.metastore.services.util:LoadHiveMetatsoreConfigsFromXmlFile help_text: | Path to a XML file containing a mapping of Hive metastore configuration key-value pairs to apply to the Hive metastore. For example: hive-site.xml hive.metastore.warehouse.dir ${test.warehouse.dir} - group: help_text: | Information used to configure the Hive metastore service as a service principal in a Kerberos realm. params: - arg_name: kerberos-principal api_field: service.hiveMetastoreConfig.kerberosConfig.principal processor: googlecloudsdk.command_lib.metastore.validators:ValidateKerberosPrincipal help_text: | A Kerberos principal that exists in the KDC to authenticate as. A typical principal is of the form "primary/instance@REALM", but there is no exact format. required: true - arg_name: krb5-config api_field: service.hiveMetastoreConfig.kerberosConfig.krb5ConfigGcsUri processor: googlecloudsdk.command_lib.metastore.validators:ValidateGcsUri:arg_name=--krb5-config help_text: | A Cloud Storage URI that specifies the path to a krb5.conf file. It is of the form gs://{bucket_name}/path/krb5.conf, although the file does not need to be named krb5.conf explicitly. required: true - arg_name: keytab api_field: service.hiveMetastoreConfig.kerberosConfig.keytab.cloudSecret processor: googlecloudsdk.command_lib.metastore.parsers:ParseSecretManagerSecretVersion help_text: | A Kerberos keytab file that can be used to authenticate a service principal with a Kerberos Key Distribution Center. This is a Secret Manager secret version, and can be fully-qualified URL, or relative name in the form `projects/{project_id}/secrets/{secret_id}/versions/{version_id}`. required: true - group: help_text: | The one hour maintenance window that specifies when Dataproc Metastore may perform system maintenance operation to the service, in UTC time. params: - arg_name: maintenance-window-day api_field: service.maintenanceWindow.dayOfWeek required: true choices: - arg_value: mon enum_value: MONDAY - arg_value: tue enum_value: TUESDAY - arg_value: wed enum_value: WEDNESDAY - arg_value: thu enum_value: THURSDAY - arg_value: fri enum_value: FRIDAY - arg_value: sat enum_value: SATURDAY - arg_value: sun enum_value: SUNDAY help_text: | The day of week when the window starts, e.g., `sun`. - arg_name: maintenance-window-hour api_field: service.maintenanceWindow.hourOfDay required: true processor: googlecloudsdk.command_lib.metastore.validators:ValidateHourOfDay help_text: | The hour of day (0-23) when the window starts. - group: help_text: | Encryption configuration for the metastore service. mutex: true params: - arg_name: encryption-kms-keys release_tracks: [ALPHA, BETA] api_field: service.encryptionConfig.kmsKeys help_text: | Comma-separated list of the Cloud KMS keys to use for customer data encryption. Cannot be changed once the service is created. Mutually exclusive with flag `--data-catalog-sync` and `--encryption-kms-key`. Each key can be provided as a fully-qualified URL, or a relative name in the form `projects/{project_id}/locations/{location}/keyRings/{key_ring_id}/cryptoKeys/{crypto_key_id}`. type: "arg_list" - arg_name: encryption-kms-key api_field: service.encryptionConfig.kmsKey processor: googlecloudsdk.command_lib.metastore.parsers:ParseCloudKmsKey help_text: | The name of the Cloud KMS key to use for customer data encryption. Cannot be changed once the service is created. Mutually exclusive with flag `--data-catalog-sync`. This can be provided as a fully-qualified URL, or a relative name in the form `projects/{project_id}/locations/{location}/keyRings/{key_ring_id}/cryptoKeys/{crypto_key_id}`. - group: help_text: | Network configuration for the metastore service. mutex: true params: - arg_name: network api_field: service.network processor: googlecloudsdk.command_lib.metastore.parsers:ParseNetwork help_text: | The name of the network on which the service can be accessed. This can be the network's ID, fully-qualified URL, or relative name in the form `projects/{project_id}/global/networks/{network}`. If "network", "consumer-subnetworks" and "network-config" are unspecified, the "default" network will be used. - arg_name: consumer-subnetworks help_text: | The list of subnetworks from which the service can be accessed. This can be the subnetwork's ID, fully-qualified URL, or relative name in the form `projects/{project_id}/regions/{region_id}/subnetworks/{subnetwork}`. type: "arg_list" - arg_name: network-config-from-file api_field: service.networkConfig type: "googlecloudsdk.calliope.arg_parsers:YAMLFileContents:" help_text: | Path to a YAML file containing the network configuration for Dataproc Metastore instance. The contents of the file should be structured as follows: YAML: consumers: - subnetwork: projects/{project_id}/regions/{region_id}/subnetworks/{subnetwork_id} - subnetwork: projects/{project_id}/regions/{region_id}/subnetworks/{subnetwork_id} ... - group: help_text: | Auxiliary versions configuration for the Dataproc Metastore service. When specified, a secondary Hive metastore service is created along with the primary service. mutex: true params: - arg_name: auxiliary-versions type: "arg_list" help_text: | Comma-separated list of auxiliary Hive metastore versions to deploy. Auxiliary Hive metastore versions must be less than the primary Hive metastore service's version. - arg_name: auxiliary-versions-from-file api_field: service.hiveMetastoreConfig.auxiliaryVersions type: "googlecloudsdk.calliope.arg_parsers:YAMLFileContents:" processor: googlecloudsdk.command_lib.metastore.services.util:LoadAuxiliaryVersionsConfigsFromYamlFile help_text: | Path to a YAML file containing the auxiliary versions configuration for Dataproc Metastore instance. The file should contain a unique auxiliary service name and auxiliary version that is lower than the primary service version. The primary version's Hive metastore configs are applied to the auxiliary version. Additional Hive metastore configs can be specified for the auxiliary version using "config_overrides". If a specified config property has already been overridden in the primary Hive metastore version's configs, the auxiliary version's override takes precedence. The contents of the file should be structured as follows: YAML: ``` - name: aux-service1 version: x.y.z config_overrides: key1: value1 key2: value2 ... - name: aux-service2 version: x.y.z config_overrides: key1: value1 key2: value2 ... ... ``` - arg_name: endpoint-protocol help_text: | The protocol to use for the metastore service endpoint. If unspecified, defaults to `THRIFT`. choices: - arg_value: thrift enum_value: THRIFT help_text: | The legacy Apache `THRIFT` protocol. - arg_value: grpc enum_value: GRPC help_text: | The modernized `GRPC` protocol. default: thrift api_field: service.hiveMetastoreConfig.endpointProtocol - group: release_tracks: [ALPHA, BETA] help_text: | Custom region configuration for the metastore service. params: - arg_name: read-write-regions api_field: service.multiRegionConfig.customRegionConfig.readWriteRegions help_text: | The list of read-write regions where the metastore service runs in. These regions should be part (or subset) of the multi-region. type: "arg_list" - arg_name: read-only-regions api_field: service.multiRegionConfig.customRegionConfig.readOnlyRegions help_text: | The list of read-only regions where the metastore service runs in. These regions should be part (or subset) of the multi-region. type: "arg_list" - group: mutex: true help_text: | Scheduled backup configurations to enable a scheduled backup of the metastore service. params: - arg_name: scheduled-backup-configs-from-file api_field: service.scheduledBackup type: "googlecloudsdk.calliope.arg_parsers:FileContents:" processor: googlecloudsdk.command_lib.metastore.services.util:LoadScheduledBackupConfigsFromJsonFile help_text: | Path to a JSON file containing a mapping of scheduled backup configurations key-value pairs to apply to the scheduled backup. For example: backup_configs.json { "enabled": true, "cron_schedule": "0 * * * *", "backup_location": "gs://example-bucket", "time_zone": "UTC" } - group: help_text: | Scheduled backup configurations to enable a scheduled backup of the metastore service. params: - arg_name: enable-scheduled-backup api_field: service.scheduledBackup.enabled action: store_true help_text: | Enable or disable scheduled backups for the Metastore service. - arg_name: scheduled-backup-cron api_field: service.scheduledBackup.cronSchedule help_text: | Specify the frequency of a scheduled backup of the metastore service. - arg_name: scheduled-backup-location api_field: service.scheduledBackup.backupLocation help_text: | Set the Cloud Storage URI to store the scheduled backups of the metastore service. - arg_name: tags api_field: service.tags.additionalProperties metavar: KEY=VALUE help_text: | List of tag KEY=VALUE pairs to add. type: arg_dict: flatten: true spec: - api_field: key - api_field: value - group: release_tracks: [ALPHA, BETA] hidden: true help_text: | Configuration required for migrating to BigQuery Metastore service. params: - arg_name: bigquery-project-id api_field: service.bigqueryMetastoreMigrationConfig.bigqueryProjectId help_text: | Project ID where the BigQuery resources (e.g. datasets, tables, etc.) should be created. required: true - arg_name: bigquery-dataset-location api_field: service.bigqueryMetastoreMigrationConfig.bigqueryDatasetLocation help_text: | The location where the BigQuery resources (e.g. datasets, tables, etc.) should be created (e.g. us-central1, us, eu, etc.). required: true - arg_name: desired-migration-state api_field: service.bigqueryMetastoreMigrationConfig.desiredMigrationState help_text: | The desired state of the migration. Note that this also reflects the current state of the migration. If an attempt to update to a new desired state fails, the migration will revert to the previous state. choices: - arg_value: migrate enum_value: MIGRATE help_text: | By setting the desired migration state to `MIGRATE`, metadata updates in Dataproc Metastore will be replicated to the BigQuery Metastore service, ensuring that it remains consistently synchronized with Dataproc Metastore. Note that this includes initial backfill of existing metadata. - arg_value: cutover enum_value: CUTOVER help_text: | By setting the desired migration state to `CUTOVER`, all metadata requests are routed to BigQuery Metastore service and Dataproc Metastore only functions as a proxy. This state can be considered as the completion of the migration. - arg_value: cancel enum_value: CANCEL help_text: | By setting the desired migration state to `CANCEL`, the migration is effectively cancelled. If the previous migration state was `MIGRATE`, then replication to BigQuery Metastore will be cancelled. If the previous state was `CUTOVER`, then metadata requests will now be served from Dataproc Metastore instead of BigQuery Metastore. Note that existing metadata changes replicated to BigQuery Metastore service are not rolled back. required: true labels: api_field: service.labels async: collection: metastore.projects.locations.operations