How to import data from cloud firestore to the local emulator?

Solution 1:

This can be accomplished through a set of commands in terminal on the existing project:

1. Login to firebase and Gcloud:

firebase login
gcloud auth login

2. See a list of your projects and connect to one:

firebase projects:list
firebase use your-project-name

gcloud projects list
gcloud config set project your-project-name

3. Export your production data to gcloud bucket with chosen name:

gcloud firestore export gs://your-project-name.appspot.com/your-choosen-folder-name

4. Now copy this folder to your local machine, I do that in functions folder directly:

cd functions
gsutil -m cp -r gs://your-project-name.appspot.com/your-choosen-folder-name .

5. Now we just want to import this folder. This should work with the basic command, thanks to latest update from Firebase team https://github.com/firebase/firebase-tools/pull/2519.

firebase emulators:start --import ./your-choosen-folder-name

Check out my article on Medium about it and a shorthanded script to do the job for you https://medium.com/firebase-developers/how-to-import-production-data-from-cloud-firestore-to-the-local-emulator-e82ae1c6ed8

Note: Its better to use a different bucket for it, as copying into your project bucket will result in the folder created in your firebase storage.
If you are interested in gsutil arguments like -m, you can see them described by executing gsutil --help.

Solution 2:

My method is somewhat manual but it does the trick. I've shared it in this useful Github thread but I'll list the steps I did here if you find them useful:

1. Go to my local Firebase project path.
2. Start the emulators using: firebase emulators:start
3. Create manually some mockup data using the GUI at http://localhost:4000/firestore using the buttons provided: + Start Collection and + Add Document.
4. Export this data locally using: emulators:export ./mydirectory
5. About the project data located at Firebase Database / Cloud Firestore, I exported a single collection like this: gcloud firestore export gs://my-project-bucket-id.appspot.com --collection-ids=myCollection The export is now located under Firebase Storage in a folder with a timestamp as name (I didn't use a prefix for my test)
6. Download this folder to local drive with: gsutil cp -r gs://my-project-bucket-id.appspot.com/myCollection ./production_data_export NOTE: I did this in a Windows environment... gsutil will throw this error: "OSError: The filename, directory name, or volume label syntax is incorrect" if the folder has invalid characters for a folder name in Windows (i.e. colons) or this error: "OSError: Invalid argument.9.0 B]" if an inner file in the folder has invalid characters too. To be able to download the export locally, rename these with a valid Windows name (i.e. removing the colons) like this: gsutil mv gs://my-project-bucket-id.appspot.com/2020-05-22T02:01:06_86152 gs://my-project-bucket-id.appspot.com/myCollection
7. Once downloaded, imitate the local export structure renaming the folder to firestore_export and copying the firebase-export-metadata.json file from the local export folder. Just to be visual, here's the structure I got:

$ tree .
.
├── local_data_export
│   ├── firebase-export-metadata.json
│   └── firestore_export
│       ├── all_namespaces
│       │   └── all_kinds
│       │       ├── all_namespaces_all_kinds.export_metadata
│       │       └── output-0
│       └── firestore_export.overall_export_metadata
└── production_data_export
    ├── firebase-export-metadata.json
    └── firestore_export
        ├── all_namespaces
        │   └── kind_myCollection
        │       ├── all_namespaces_kind_myCollection.export_metadata
        │       ├── output-0
        │       └── output-1
        └── firestore_export.overall_export_metadata

8 directories, 9 files

8. Finally, start the local emulator pointing to this production data to be imported: firebase emulators:start --import=./mock_up_data/production_data_export/
9. You should see the imported data at: http://localhost:4000/firestore/

This should assist readers for now, while we await a more robust solution from the Firebase folks.

Solution 3:

You can use the firestore-backup-restore to export and import your production data as JSON files.

I wrote a quick hack to allow for importing these JSON in the Firebase Simulator Firestore instance.

I proposed a pull request and made this npm module in the meantime.

You can use it this way:

const firestoreService = require('@crapougnax/firestore-export-import')
const path = require('path')

// list of JSON files generated with the export service
// Must be in the same folder as this script
const collections = ['languages', 'roles']

// Start your firestore emulator for (at least) firestore
// firebase emulators:start --only firestore

// Initiate Firebase Test App
const db = firestoreService.initializeTestApp('test', {
   uid: 'john',
   email: '[email protected]',
})

// Start importing your data
let promises = []
try {
   collections.map(collection =>
      promises.push(
         firestoreService.fixtures(
            path.resolve(__dirname, `./${collection}.json`),
            [],
            [],
            db,
         ),
      ),
   )
   Promise.all(promises).then(process.exit)
} catch (err) {
   console.error(err)
}

Obviously, since this data won't persist in the emulator, you'll typically inject them in the before() function of your test suite or even before every test.