MongoDB: Sharding Concepts

Sharding Topology:

Applications will always talk to Mongos server, Mongos server is aware of the data distribution and  fetches the information from config servers.

And accordingly route the session to respective shard.

Lets do the setup:


1.) Config server setup,

[mongod@sikki4u2c sharding]$ pwd
/home/mongod/sharding
[mongod@sikki4u2c sharding]$ mkdir -p ConfigShard1/config1_srv_1
                                                     mkdir -p ConfigShard1/config1_srv_2
                                                     mkdir -p ConfigShard1/config1_srv_3
[mongod@sikki4u2c sharding]$ ll
total 0
drwxrwxr-x. 2 mongod mongod 6 Sep  9 05:49 config1
[mongod@sikki4u2c sharding]$   mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_1 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_1.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 2541
child process started successfully, parent exiting
[mongod@sikki4u2c sharding]$

[mongod@sikki4u2c ConfigShard1]$ mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_2 --port 20002 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_2.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 3172
child process started successfully, parent exiting
[mongod@sikki4u2c ConfigShard1]$ mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_1 --port 20001 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_1.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 3208
child process started successfully, parent exiting
[mongod@sikki4u2c ConfigShard1]$ mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_3 --port 20003 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_3.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 3297
child process started successfully, parent exiting
[mongod@sikki4u2c ConfigShard1]$

Add them to replicaset.

2.) Now lets start Mongos our sharding router process.

This process needs to know one thing , i,e. where is the config server.

mongos --configdb replSetConfig01/localhost:20001,localhost:20002,localhost:20003 --logpath /home/mongod/sharding/ConfigShard1/shard_01.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 3719
child process started successfully, parent exiting
[mongod@sikki4u2c ConfigShard1]$

3.) Lets configure the mongod server with actual data.

Lets configure replica set 1

mongod --dbpath /home/mongod/sharding/ConfigShard1/mongod_data_01 --port 30001 --replSet r1 --logpath /home/mongod/sharding/ConfigShard1/mongod_data_01/mongod_m1.log --fork
mongod --dbpath /home/mongod/sharding/ConfigShard1/mongod_data_02 --port 30002 --replSet r1 --logpath /home/mongod/sharding/ConfigShard1/mongod_data_02/mongod_m2.log --fork

Lets setup another replica set 

mongod --dbpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr1_rs2 --port 30003 --replSet r2 --logpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr1_rs2/mongod.log --fork

mongod --dbpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr2_rs2 --port 30004 --replSet r2 --logpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr2_rs2/mongod.log --fork

Done:

Till now we have configured 

a config server replica set,

a shard server

and 2 mongod replicaset with actual data

mongod    3172     1  0 06:09 ?        00:00:05 mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_2 --port 20002 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_2.log --fork

mongod    3208     1  0 06:10 ?        00:00:05 mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_1 --port 20001 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_1.log --fork

mongod    3297     1  0 06:10 ?        00:00:05 mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_3 --port 20003 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_3.log --fork

mongod    3719     1  0 06:17 ?        00:00:00 mongos --configdb replSetConfig01/localhost:20001,localhost:20002,localhost:20003 --logpath /home/mongod/sharding/ConfigShard1/shard_01.log --fork

mongod    3820     1  0 06:21 ?        00:00:02 mongod --dbpath /home/mongod/sharding/ConfigShard1/mongod_data_01 --port 30001 --replSet r1 --logpath /home/mongod/sharding/ConfigShard1/mongod_data_01/mongod_m1.log --fork

mongod    3884     1  0 06:22 ?        00:00:02 mongod --dbpath /home/mongod/sharding/ConfigShard1/mongod_data_02 --port 30002 --replSet r1 --logpath /home/mongod/sharding/ConfigShard1/mongod_data_02/mongod_m2.log --fork

mongod    4153     1  0 06:26 ?        00:00:00 mongod --dbpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr1_rs2 --port 30003 --replSet r2 --logpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr1_rs2/mongod.log --fork

mongod    4183     1  0 06:26 ?        00:00:00 mongod --dbpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr2_rs2 --port 30004 --replSet r2 --logpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr2_rs2/mongod.log --fork

mongod    4378  2336  0 06:28 pts/0    00:00:00 ps -ef

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

In sharded env, anything / mostly you do will be done through sharded server. (mongos), as it is the one that knows

For us mongos is running on default port 27017, so we connec to mongos

mongo
mongos> sh.addShard('r1/localhost:30001,localhost:30002')
{
        "shardAdded" : "r1",
        "ok" : 1,
        "operationTime" : Timestamp(1568052737, 4),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1568052737, 4),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
mongos>

mongos> sh.addShard('r2/localhost:30003,localhost:30004')

{

        "shardAdded" : "r2",

        "ok" : 1,

        "operationTime" : Timestamp(1568052808, 2),

        "$clusterTime" : {

                "clusterTime" : Timestamp(1568052808, 2),

                "signature" : {

                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

                        "keyId" : NumberLong(0)

                }

        }

}

mongos>

No i will use below code to create demo data

for (var i = 1; i <= 100000; i++) {
   db.testData.insert( { x : i } )
}

Then i need to enable sharding on database, my collection testData is in Demo database.

mongos> sh.enableSharding('demo');
{
        "ok" : 1,
        "operationTime" : Timestamp(1568053819, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1568053819, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

Lets check the cluster status

mongos> sh.status()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("5d75ed206988427493d216bc")
  }
  shards:
        {  "_id" : "r1",  "host" : "r1/localhost:30001,localhost:30002",  "state" : 1 }
        {  "_id" : "r2",  "host" : "r2/localhost:30003,localhost:30004",  "state" : 1 }
  active mongoses:
        "3.6.14" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                No recent migrations
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                r1      1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : r1 Timestamp(1, 0)
        {  "_id" : "demo",  "primary" : "r2",  "partitioned" : true }

mongos>

Notice the line in bold.

Now lets shard our collection, on "x" field

mongos> sh.shardCollection('demo.testData', {'x':1})

{

        "collectionsharded" : "demo.testData",

        "collectionUUID" : UUID("ff66a1f3-6384-416c-9b11-bb74b1b9a30a"),

        "ok" : 1,

        "operationTime" : Timestamp(1568054173, 13),

        "$clusterTime" : {

                "clusterTime" : Timestamp(1568054173, 13),

                "signature" : {

                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

                        "keyId" : NumberLong(0)

                }

        }

}

mongos> show dbs

admin   0.000GB

config  0.001GB

demo    0.002GB

mongos> use demo

switched to db demo

mongos> show collections

testData

mongos>

i did not created a database "demo" in mongos, i added to shard, so we can see the db in mongos now

mongos> use config

switched to db config

mongos>

mongos> db.chunks.find().pretty()

{

        "_id" : "config.system.sessions-_id_MinKey",

        "ns" : "config.system.sessions",

        "min" : {

                "_id" : { "$minKey" : 1 }

        },

        "max" : {

                "_id" : { "$maxKey" : 1 }

        },

        "shard" : "r1",

        "lastmod" : Timestamp(1, 0),

        "lastmodEpoch" : ObjectId("5d76964ec058b369d1d2e6f0")

} --- this chunk associate itself with ReplicaSet "r1" and it has the min and max key

{

        "_id" : "demo.testData-x_MinKey",

        "ns" : "demo.testData",

        "min" : {

                "x" : { "$minKey" : 1 }

        },

        "max" : {

                "x" : { "$maxKey" : 1 }

        },

        "shard" : "r2",

        "lastmod" : Timestamp(1, 0),

        "lastmodEpoch" : ObjectId("5d769b9dc058b369d1d300fc")

}--- this chunk associate itself with ReplicaSet "r2" and it has the min and max key

mongos>