A new check on MongoDB databases on Compose will reveal if you've placed data in a bad place. In this article, we'll show you how to get it to the good place.
If you have a MongoDB database, you may be seeing this warning on the overview -
Warning: Found unexpected data stored in admin database.
If you aren't seeing the warning, good news, you aren't affected and your data is already in the good place. If you haven't checked your MongoDB's overview on the web recently, we recommend you check it. If you see this message, read on.
We've written in the past about the dangers of leaving data in the admin database within MongoDB. The short version is that it is not somwhere you want to keep data. The admin database is an auxiliary database within MongoDB for, as the name suggests, administration and it should only hold admin user credentials and some metadata. Keeping your actual data in there is bad practice, it's bad for performance and bad for reliability.
We recommend you move it out as soon as possible and make sure your applications aren't writing there.
Finding out what's in there
Create yourself an admin level user, if you haven't already. You can do this through the Compose console's browser by accessing the admin database. Users created here get almost complete control of the database. You'll notice you can't see any collections in the Compose console browser for the admin collection - you'll have to log in with
mongo shell to continue. You can find an example command line on the Compose console overview for your database. Just replace the values after
-p with your user's username and password. Then log in:
MongoDB shell version v3.4.9 connecting to: mongodb://portal-ssl980-0.wiredfortigers.compose-3.composedb.com:10980/compose MongoDB server version: 3.2.10 WARNING: shell and server versions do not match mongos>
Once logged in, ask to use the admin database.
mongos> use admin switched to db admin
And then list the collections saved in the admin database using
mongos> db.getCollectionNames() [ "system.users", "system.version", "words" ] mongos>
And there's the troublemaker. The
system. collections are meant to be there but someone's created a
words collection in admin.
Is the collection used?
In our example, we know our words collection isn't used and we could fix the warning by simply dropping the collection. Non-system collections in admin aren't backed up so you'll want to be completely sure that the collection or collections are not used before doing this. There is no way back. You may have an application that is relying on this collection, so before you commit to it, let's make a copy of the data.
Copying a collection
We're going to make a copy of the collection. To assist in this, we are also providing a script, copyCollection.sh which breaks out all the settings. If you browse the MongoDB documentation, you'll find a number of functions for copying collections. Unfortunately they either don't work with a sharded MongoDB or mongos, as provided on Compose, or they are deprecated because they bring the database to a halt while the copying takes place. So, what we have here is a script which uses
mongoexport to export a collection from
admin. It then pipes the results straight into
mongoimport which writes it to a new collection in the
# MONGOHOST is the host the database is accessible on - include hostname and port export MONGOHOST=portal-ssl980-0.wiredfortigers.compose-3.composedb.com:10980 # COLLECTION is the name of the collection we want to copy from in the admin database export COLLECTION=words # NEWCOLLECTION is the name of the collection we want to create in the compose database export NEWCOLLECTION=wordstoo # MONGOCREDENTIALS is the settings for user, password and authentication database # where that user can be found - note this should be a user in the admin database with # sufficient privileges to access both collections in each database. export MONGOCREDENTIALS="-u username -p password --authenticationDatabase admin" # SSLPARAMS enables SSL with no certificate verification - leave empty for no SSL export SSLPARAMS="--ssl --sslAllowInvalidCertificates" # This command now exports from admin.$COLLECTION and writes it to compose.$NEWCOLLECTION # It does not delete the original collection in admin mongoexport $SSLPARAMS --host $MONGOHOST $MONGOCREDENTIALS --db admin --collection $COLLECTION | mongoimport $SSLPARAMS --host $MONGOHOST $MONGOCREDENTIALS --db compose --collection $NEWCOLLECTION
All you need to do is pop in the correct host and port in
MONGOHOST, name of the collection you want to copy from (
COLLECTION), name of the new collection to be created (
NEWCOLLECTION) and substitute in your administration user and password in
MONGOCREDENTIALS. Then run the script
bash copyCollection.sh 2017-12-11T12:10:51.846+0000 connected to: portal-ssl980-0.wiredfortigers.compose-3.composedb.com:10980 2017-12-11T12:10:51.985+0000 connected to: portal-ssl980-0.wiredfortigers.compose-3.composedb.com:10980 2017-12-11T12:10:52.074+0000 exported 3 records 2017-12-11T12:10:52.765+0000 imported 3 documents
If we now log in on the command line, we can see our new collection:
mongos> use compose switched to db compose mongos> db.getCollectionNames() [ "wordstoo" ] mongos>
And, after checking everything is fine, go back to the admin database and drop the old collection.
mongos> use admin switched to db admin mongos> db.words.drop() true mongos>
There will be a delay between this being done and the warning in the Compose console going away due to the way this check is performed. You'll have to wait up to an hour before the warning clears from the Compose UI and API.
Once those collections have been dropped, do check back after a day or so to ensure none of your applications are still creating collections in the
admin database. If they are, you'll need to correct their connection code and, for safety, explicitly use the drivers implementation of
use databasename to address the right, non-admin database.
The important thing to take away from this process is that moving your collections out of the
admin database means your data will be safe and in the right part of MongoDB. That also means your data is then being backed up correctly and performing optimally. Take note of the warning and act on it today to protect your data.
Read more articles about Compose databases - use our Curated Collections Guide for articles on each database type. If you have any feedback about this or any other Compose article, drop the Compose Articles team a line at firstname.lastname@example.org. We're happy to hear from you.
attribution Ash from Modern Afflatus