Mongo DB Journal: September 2019

Saturday, September 14, 2019

MongoDB: Security

db.system.users.find()

db.system.users.find({},{credentials:0}).pretty()

to list all users.

Friday, September 13, 2019

MOngoDB : Monitoring

1.) Logfile: /var/log/mongod/mongo.log or you can also the logs remotly from below commands.

r1:PRIMARY> show logs
global
rs
startupWarnings
r1:PRIMARY> show logs startupWarnings
global
rs
startupWarnings
r1:PRIMARY> show log startupWarnings
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten]
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten]
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** WARNING: This server is bound to localhost.
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** Remote systems will be unable to connect to this server.
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** Start the server with --bind_ip <address> to specify which IP
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** addresses it should serve responses from, or with --bind_ip_all to
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** bind to all interfaces. If this behavior is desired, start the
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disable this warning.
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten]
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten]
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten]
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2019-09-10T08:53:54.730+0000 I CONTROL [initandlisten]
r1:PRIMARY>

2.) verbosity: if you wna tto see details logs you can set the level of verbosity.
Verbosity level numberic 0-5
Level Description
F Fatal
E Error
W Warning
I Informational, for Verbosity Level of 0
D[1-5] Debug, for All Verbosity Levels > 0

you can also set the verbosity of specific topic instead of everything
(accesscontrol, command, control, geo, index, network,query,replication,storage, journal, write)

example: Syntax: db.setLogLevel(0,'query')
db.setLogLevel(0,'index') OR db.setLogLevel(0,'replication')

note: if you dont supply a topic, by default it will enable logging for everything.

Lets see the verbosity 4 for query component.

r1:PRIMARY> db.setLogLevel(4,'query')
{
"was" : {
"verbosity" : 0, ---- this was before
"accessControl" : {
"verbosity" : -1
},
"command" : {
"verbosity" : -1
},
"control" : {
"verbosity" : -1
},
"executor" : {
"verbosity" : -1
},
"geo" : {
"verbosity" : -1
},
"index" : {
"verbosity" : -1
},
"network" : {
"verbosity" : -1,
"asio" : {
"verbosity" : -1
},
"bridge" : {
"verbosity" : -1
}
},
"query" : {
"verbosity" : -1
},
"replication" : {
"verbosity" : -1,
"heartbeats" : {
"verbosity" : -1
},
"rollback" : {
"verbosity" : -1
}
},
"sharding" : {
"verbosity" : -1,
"shardingCatalogRefresh" : {
"verbosity" : -1
}
},
"storage" : {
"verbosity" : -1,
"journal" : {
"verbosity" : -1
}
},
"write" : {
"verbosity" : -1
},
"ftdc" : {
"verbosity" : -1
},
"tracking" : {
"verbosity" : -1
}
},
"ok" : 1,
"operationTime" : Timestamp(1568112427, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1568112427, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
r1:PRIMARY>

It should have changed of the verbosity of query, lets check

r1:PRIMARY> db.getLogComponents()
{
"verbosity" : 0,
"accessControl" : {
"verbosity" : -1
},
"command" : {
"verbosity" : -1
},
"control" : {
"verbosity" : -1
},
"executor" : {
"verbosity" : -1
},
"geo" : {
"verbosity" : -1
},
"index" : {
"verbosity" : -1
},
"network" : {
"verbosity" : -1,
"asio" : {
"verbosity" : -1
},
"bridge" : {
"verbosity" : -1
}
},
"query" : {
"verbosity" : 4 --- query verbosity changed to 4
},
"replication" : {
"verbosity" : -1,
"heartbeats" : {
"verbosity" : -1
},
"rollback" : {
"verbosity" : -1
}
},
"sharding" : {
"verbosity" : -1,
"shardingCatalogRefresh" : {
"verbosity" : -1
}
},
"storage" : {
"verbosity" : -1,
"journal" : {
"verbosity" : -1
}
},
"write" : {
"verbosity" : -1
},
"ftdc" : {
"verbosity" : -1
},
"tracking" : {
"verbosity" : -1
}

Lets see of log is created

r1:PRIMARY> use demo

switched to db demo

r1:PRIMARY> db.testData.count()

100000

r1:PRIMARY> show logs

global

startupWarnings

r1:PRIMARY> show log global

2019-09-10T10:47:08.497+0000 I COMMAND [conn21] successfully set parameter logComponentVerbosity to { query: { verbosity: 4.0 } } (was { verbosity: 0, accessControl: { verbosity: -1 }, command: { verbosity: -1 }, control: { verbosity: -1 }, executor: { verbosity: -1 }, geo: { verbosity: -1 }, index: { verbosity: -1 }, network: { verbosity: -1, asio: { verbosity: -1 }, bridge: { verbosity: -1 } }, query: { verbosity: -1 }, replication: { verbosity: -1, heartbeats: { verbosity: -1 }, rollback: { verbosity:

2019-09-10T10:48:54.741+0000 D QUERY [LogicalSessionCacheRefresh] Using idhack: { _id: { id: UUID("b943e79b-fb79-477f-a5b7-2057ac5447c1"), uid: BinData(0, E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855) } }

2019-09-10T10:48:54.746+0000 D QUERY [LogicalSessionCacheReap] Only one plan is available; it will be run but will not be cached. query: { lastWriteDate: { $lt: new Date(1568110734741) } } sort: { _id: 1 } projection: { _id: 1 }, planSummary: IXSCAN { _id: 1 }

2019-09-10T10:53:54.741+0000 D QUERY [LogicalSessionCacheRefresh] Using idhack: { _id: { id: UUID("b943e79b-fb79-477f-a5b7-2057ac5447c1"), uid: BinData(0, E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855) } }

2019-09-10T10:53:54.746+0000 D QUERY [LogicalSessionCacheReap] Only one plan is available; it will be run but will not be cached. query: { lastWriteDate: { $lt: new Date(1568111034741) } } sort: { _id: 1 } projection: { _id: 1 }, planSummary: IXSCAN { _id: 1 }

r1:PRIMARY>

See this, some extra information is written about query execution.

to unset this, u need set the verbosity to -1.

db.setLogLevel(-1,'query')

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

you can also verbosity in config file, if you want to set that it should be there after a reboot.

Say i want to set verbosity level 2 for command topic.

it needs to added under

systemLog:

component:

command:

verbosity: 2

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

3: Query Profiler: it can capture slow query so you can erview later

Mongo has 3 profiling levels, 0,1 & 2

0 is default. it means no profiling data is collected at all,

1 only slow operations are logged

2 all operations are logged regardless of their speed.

In level 1, you can supply a threshold in milliseconds, operations that take longer than that threshold are logged, faster operations are not.

lets enbale profiling for any operations running more than 2 millisencons

r1:PRIMARY> show profile

db.system.profile is empty

Use db.setProfilingLevel(2) will enable profiling

Use db.system.profile.find() to show raw profile entries

r1:PRIMARY> db.setProfilingLevel(1,2)

{

"was" : 0,

"slowms" : 100,

"sampleRate" : 1,

"ok" : 1,

"operationTime" : Timestamp(1568113657, 1),

"$clusterTime" : {

"clusterTime" : Timestamp(1568113657, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
db.system.profile:
from lot of profiling data, you can look for query specific to your collections
operation type: "Query"
namespace: is the collections name

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

4. Mongostat: lets you see metrics on several key things. Mongostat is part of the

[mongod@sikki4u2c ~]$ mongostat --port 30001
insert query update delete getmore command dirty used flushes vsize res qrw arw net_in net_out conn set repl time
*0 *0 *0 *0 2 4|0 0.0% 0.0% 0 1.41G 79.0M 0|0 1|0 2.20k 65.1k 14 r1 PRI Sep 13 08:53:55.349
*0 *0 *0 *0 0 2|0 0.0% 0.0% 0 1.41G 79.0M 0|0 1|0 416b 63.1k 14 r1 PRI Sep 13 08:53:56.350
*0 *0 *0 *0 0 1|0 0.0% 0.0% 0 1.41G 79.0M 0|0 1|0 157b 62.5k 14 r1 PRI Sep 13 08:53:57.351
*0 *0 *0 *0 0 4|0 0.0% 0.0% 0 1.41G 79.0M 0|0 1|0 697b 63.9k 14 r1 PRI Sep 13 08:53:58.349
*0 *0 *0 *0 0 1|0 0.0% 0.0% 0 1.41G 79.0M 0|0 1|0 157b 62.5k 14 r1 PRI Sep 13 08:53:59.349
*0 *0 *0 *0 0 3|0 0.0% 0.0% 0 1.41G 79.0M 0|0 1|0 1.43k 64.0k 14 r1 PRI Sep 13 08:54:00.350

^C2019-09-13T08:54:01.253+0000 signal 'interrupt' received; forcefully terminating

insert update delete : Each one is the no of operations per second from the client. our is very quite system. nothing running. so they are all zero.

getmore : when there is a cursor and the client asks for more data on an open cursor.

query : it could be lower than the "getmore" in case you have not so many queries but each query is returning many many documents. wen there is too many documetns and thewy dont just fit in the fisrt batch of the cursor then the applicatio will be issuing another "getmore" and another "getmore" for the same query, there fore the query number might be lower than the "getmore's"

flush: is the no of disk flushes per second. "flush" relates to I/O, high flush rates means more I/O pressure.

res: is the no of megabytes of resident memore, roughly this is how much memory the mongod process consumes. this number can grow over time and can approach the avaialbel ram, this is expected, after all mongo uses available memory to speed up operations, another reson to tun mongo on its own server.

mapped: is megabytes of data that is mapped from memory mapped files , this is relevant with the "mmapv1" the classic memory mapped file storage engine. this number can far exceed your availabel RAM size which is fine, this indications tells you how much of the data on disk was loaded at some point by mongo.

vsize: is the amount of virtual memory the mongod process is using. f

faults: IS the no of page faults per second, when data is required and is not in memory yet, it has to be brought in to the disk, while we do expect that mongo needs data from disk, high spikes here or large ramp up should be investigated. this number can go up due to no of factors, if your system starts to get bugged down by I/O this number can often be an early warning

idx miss: refers to the percentage of attempts to load an index part that is not in memory, causing a page fault. we would like this no to be low. if the parts of the index needed overtime dont fit in memory, this number can grow. same concerns as page faults. In general IO is the slowest part, whatever you can do to minimize IO you should.

qrw arw : are the no of queued reads and writes. mongo only runs certain no of total reads and writes in given time. if more are requested by clients , those requests are queued untill current read and write complete. high queuing means high demand, if mongo cant keep up then the no of queued operations will spike.

ar aw: are the no of active reads and wrties, these are the non queued ones. looking at these and queued ones help u see if capacity is appropriate. if u see a high no of queued operations u might need to scale out or up. the no of active ops gives u a sense of how many ur server can handle because if it could handle more , the no of actives would be higher and the no of queued will lower.

net in net out : is the bytes on th network. gives us a sense of overall traffic on the wire.

conn: this number includes all connections from other mongo replica member, application connection and etc. Connections needs to controlled, a very high no of connections would result in high memory usage.

set: shows the replica set name

repl: shows the role of member being monitored.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
5.) MongoTop : gives overall engine stats. It shows where mongo spends most of its time. it shows time spent doing reads and writes in the most active collections

[mongod@sikki4u2c ~]$ mongotop --port 30001
2019-09-13T16:20:55.590+0000 connected to: 127.0.0.1:30001

ns total read write 2019-09-13T16:20:56Z
admin.system.keys 0ms 0ms 0ms
admin.system.roles 0ms 0ms 0ms
admin.system.version 0ms 0ms 0ms
config.system.sessions 0ms 0ms 0ms
config.transactions 0ms 0ms 0ms
demo.testData 0ms 0ms 0ms
demo.testData_1 0ms 0ms 0ms
local.me 0ms 0ms 0ms
local.oplog.rs 0ms 0ms 0ms
local.replset.election 0ms 0ms

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

db.stats() : using this fucntion we can get the disk and memory usage estimates.

connect to any database:

r1:PRIMARY> use demo
switched to db demo
r1:PRIMARY> db.stats()
{
"db" : "demo",
"collections" : 2,
"views" : 0,
"objects" : 100020,
"avgObjSize" : 33,
"dataSize" : 3300660, --- in bytes
"storageSize" : 1691648,
"numExtents" : 0,
"indexes" : 2,
"indexSize" : 954368,
"fsUsedSize" : 10444038144,
"fsTotalSize" : 21463281664,
"ok" : 1,
"operationTime" : Timestamp(1568391949, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1568391949, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}

to see in megabye

db.stats(1000000)

++++++++++++++++++++++++++++++++++++++++++

This picture is what we studied: see the flow here:

1. using mongotop we found where mongo was spending lot of time,
2 using db.stats() on the collection in question we found some fishy index that may be not so usefull. removing this index reduced the size of indx on disk , reduces the work mongo has to do when writing to the collection and conserve resource.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6.) db.serverStatus() : gather the mix of runtime statistics and resource usgae. Mongo give statistics and numbers resource usage, locks, timings, =memory, data size, flush intervals, etc.

https://docs.mongodb.com/manual/reference/command/serverStatus/

entire output of db.serverStatus() can be too much, you can use specific component command
db.serverStatus().dur --durability info
db.serverStatus().mem -- memory statistcis
db.serverStatus().network --gives network statistics
db.serverStatus().metrics.cursor -- related to queries -- cursor gives indication of open cursor
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

mongostat --port 30001 --rowcount 3

tell mongo just to print 3 rows n then exit.

Tuesday, September 10, 2019

MongoDB: Sharding Concepts

Sharding Topology:

Applications will always talk to Mongos server, Mongos server is aware of the data distribution and fetches the information from config servers.

And accordingly route the session to respective shard.

Lets do the setup:

1.) Config server setup,

[mongod@sikki4u2c sharding]$ pwd
/home/mongod/sharding
[mongod@sikki4u2c sharding]$ mkdir -p ConfigShard1/config1_srv_1
mkdir -p ConfigShard1/config1_srv_2
mkdir -p ConfigShard1/config1_srv_3
[mongod@sikki4u2c sharding]$ ll
total 0
drwxrwxr-x. 2 mongod mongod 6 Sep 9 05:49 config1
[mongod@sikki4u2c sharding]$ mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_1 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_1.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 2541
child process started successfully, parent exiting
[mongod@sikki4u2c sharding]$

[mongod@sikki4u2c ConfigShard1]$ mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_2 --port 20002 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_2.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 3172
child process started successfully, parent exiting
[mongod@sikki4u2c ConfigShard1]$ mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_1 --port 20001 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_1.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 3208
child process started successfully, parent exiting
[mongod@sikki4u2c ConfigShard1]$ mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_3 --port 20003 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_3.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 3297
child process started successfully, parent exiting
[mongod@sikki4u2c ConfigShard1]$

Add them to replicaset.

2.) Now lets start Mongos our sharding router process.

This process needs to know one thing , i,e. where is the config server.

mongos --configdb replSetConfig01/localhost:20001,localhost:20002,localhost:20003 --logpath /home/mongod/sharding/ConfigShard1/shard_01.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 3719
child process started successfully, parent exiting
[mongod@sikki4u2c ConfigShard1]$

3.) Lets configure the mongod server with actual data.

Lets configure replica set 1

mongod --dbpath /home/mongod/sharding/ConfigShard1/mongod_data_01 --port 30001 --replSet r1 --logpath /home/mongod/sharding/ConfigShard1/mongod_data_01/mongod_m1.log --fork
mongod --dbpath /home/mongod/sharding/ConfigShard1/mongod_data_02 --port 30002 --replSet r1 --logpath /home/mongod/sharding/ConfigShard1/mongod_data_02/mongod_m2.log --fork

Lets setup another replica set

mongod --dbpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr1_rs2 --port 30003 --replSet r2 --logpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr1_rs2/mongod.log --fork

mongod --dbpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr2_rs2 --port 30004 --replSet r2 --logpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr2_rs2/mongod.log --fork

Done:

Till now we have configured

a config server replica set,

a shard server

and 2 mongod replicaset with actual data

mongod 3172 1 0 06:09 ? 00:00:05 mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_2 --port 20002 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_2.log --fork

mongod 3208 1 0 06:10 ? 00:00:05 mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_1 --port 20001 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_1.log --fork

mongod 3297 1 0 06:10 ? 00:00:05 mongod --configsvr --dbpath /home/mongod/sharding/ConfigShard1/config1_srv_3 --port 20003 --replSet replSetConfig01 --logpath /home/mongod/sharding/ConfigShard1/config_srv_3.log --fork

mongod 3719 1 0 06:17 ? 00:00:00 mongos --configdb replSetConfig01/localhost:20001,localhost:20002,localhost:20003 --logpath /home/mongod/sharding/ConfigShard1/shard_01.log --fork

mongod 3820 1 0 06:21 ? 00:00:02 mongod --dbpath /home/mongod/sharding/ConfigShard1/mongod_data_01 --port 30001 --replSet r1 --logpath /home/mongod/sharding/ConfigShard1/mongod_data_01/mongod_m1.log --fork

mongod 3884 1 0 06:22 ? 00:00:02 mongod --dbpath /home/mongod/sharding/ConfigShard1/mongod_data_02 --port 30002 --replSet r1 --logpath /home/mongod/sharding/ConfigShard1/mongod_data_02/mongod_m2.log --fork

mongod 4153 1 0 06:26 ? 00:00:00 mongod --dbpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr1_rs2 --port 30003 --replSet r2 --logpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr1_rs2/mongod.log --fork

mongod 4183 1 0 06:26 ? 00:00:00 mongod --dbpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr2_rs2 --port 30004 --replSet r2 --logpath /home/mongod/sharding/ConfigShard1/Shard_RS2/mongo_svr2_rs2/mongod.log --fork

mongod 4378 2336 0 06:28 pts/0 00:00:00 ps -ef

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

In sharded env, anything / mostly you do will be done through sharded server. (mongos), as it is the one that knows

For us mongos is running on default port 27017, so we connec to mongos

mongo
mongos> sh.addShard('r1/localhost:30001,localhost:30002')
{
"shardAdded" : "r1",
"ok" : 1,
"operationTime" : Timestamp(1568052737, 4),
"$clusterTime" : {
"clusterTime" : Timestamp(1568052737, 4),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
mongos>

mongos> sh.addShard('r2/localhost:30003,localhost:30004')

{

"shardAdded" : "r2",

"ok" : 1,

"operationTime" : Timestamp(1568052808, 2),

"$clusterTime" : {

"clusterTime" : Timestamp(1568052808, 2),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

mongos>

No i will use below code to create demo data

for (var i = 1; i <= 100000; i++) {
db.testData.insert( { x : i } )
}

Then i need to enable sharding on database, my collection testData is in Demo database.

mongos> sh.enableSharding('demo');
{
"ok" : 1,
"operationTime" : Timestamp(1568053819, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1568053819, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}

Lets check the cluster status

mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5d75ed206988427493d216bc")
}
shards:
{ "_id" : "r1", "host" : "r1/localhost:30001,localhost:30002", "state" : 1 }
{ "_id" : "r2", "host" : "r2/localhost:30003,localhost:30004", "state" : 1 }
active mongoses:
"3.6.14" : 1
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "config", "primary" : "config", "partitioned" : true }
config.system.sessions
shard key: { "_id" : 1 }
unique: false
balancing: true
chunks:
r1 1
{ "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : r1 Timestamp(1, 0)
{ "_id" : "demo", "primary" : "r2", "partitioned" : true }

mongos>

Notice the line in bold.

Now lets shard our collection, on "x" field

mongos> sh.shardCollection('demo.testData', {'x':1})

{

"collectionsharded" : "demo.testData",

"collectionUUID" : UUID("ff66a1f3-6384-416c-9b11-bb74b1b9a30a"),

"ok" : 1,

"operationTime" : Timestamp(1568054173, 13),

"$clusterTime" : {

"clusterTime" : Timestamp(1568054173, 13),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

mongos> show dbs

admin 0.000GB

config 0.001GB

demo 0.002GB

mongos> use demo

switched to db demo

mongos> show collections

testData

mongos>

i did not created a database "demo" in mongos, i added to shard, so we can see the db in mongos now

mongos> use config

switched to db config

mongos>

mongos> db.chunks.find().pretty()

{

"_id" : "config.system.sessions-_id_MinKey",

"ns" : "config.system.sessions",

"min" : {

"_id" : { "$minKey" : 1 }

"max" : {

"_id" : { "$maxKey" : 1 }

"shard" : "r1",

"lastmod" : Timestamp(1, 0),

"lastmodEpoch" : ObjectId("5d76964ec058b369d1d2e6f0")

} --- this chunk associate itself with ReplicaSet "r1" and it has the min and max key

{

"_id" : "demo.testData-x_MinKey",

"ns" : "demo.testData",

"min" : {

"x" : { "$minKey" : 1 }

"max" : {

"x" : { "$maxKey" : 1 }

"shard" : "r2",

"lastmod" : Timestamp(1, 0),

"lastmodEpoch" : ObjectId("5d769b9dc058b369d1d300fc")

}--- this chunk associate itself with ReplicaSet "r2" and it has the min and max key

mongos>

MOngoDb : Sharding Keys

Always pick the field with High Cardinality as shard key, so that the data is distributed well among all shard servers.

Sharding can be based on Chunks defines keys and tags defines keys.

Chunks defines keys :
the shard key you pick should be such that it provides a god variety over all documents. Ideally we should always be able to take a chunk , a range of keys and split them futhur as more documetns are added. Mongo will automatically split the chunks in smaller chunk when it feels a chunk is too large.

Pic1: Large chunk: Minkey 1 Max key 4 Pic 2: Chunk split into 2 chunk, 1-2,2-4

Tags defines keys:
lets you designate a shard for a range of keys . Lets consider where we have one shard in a DataCenter in Amstermdam, other shard in NewZEaland. The latency will be increased. If would be cool so say mongo, if you see a phone prefix from any european country, those should be spread among the shard in Amsterdam DC.

Say the phone number data, all country with european phone prefix goes to Europe, and American phone prefix in american shard.

Lets say if below exmaple, the applications send a data, mongo inspects the shard key value in that documetns. It finds a tag range defination that covers that shard key value. Using the tag name it will locate a chunk that lives on one of the shards that is marked with the tag. If a chunk dosent exists, it will create one on of the shards that has a specified tag.

Example:

mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5d75ed206988427493d216bc")
}
shards:
{ "_id" : "r1", "host" : "r1/localhost:30001,localhost:30002", "state" : 1 }
{ "_id" : "r2", "host" : "r2/localhost:30003,localhost:30004", "state" : 1 }
active mongoses:
"3.6.14" : 1

Lets tag them

mongos> sh.addShardTag('r1','europe')

{

"ok" : 1,

"operationTime" : Timestamp(1568108258, 1),

"$clusterTime" : {

"clusterTime" : Timestamp(1568108258, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

mongos> sh.addShardTag('r2','america')

{

"ok" : 1,

"operationTime" : Timestamp(1568108273, 2),

"$clusterTime" : {

"clusterTime" : Timestamp(1568108273, 2),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

mongos>

mongos> sh.status()

--- Sharding Status ---

sharding version: {

"_id" : 1,

"minCompatibleVersion" : 5,

"currentVersion" : 6,

"clusterId" : ObjectId("5d75ed206988427493d216bc")

}

shards:

{ "_id" : "r1", "host" : "r1/localhost:30001,localhost:30002", "state" : 1, "tags" : [ "europe" ] }

{ "_id" : "r2", "host" : "r2/localhost:30003,localhost:30004", "state" : 1, "tags" : [ "america" ] }

active mongoses:

"3.6.14" : 1

Creating a tag range: it defines a range that is assigned to a tag, helps balancer to automatically assign chunks to desired shards.

I below exmaple, in messages collections, i want my country prefix '1' and '2' to live in the shard america. SO lower bound is 1 and upper bound is country prefix 2.
Note: Lower bound in inclusice, upper bound is exclusive. So upper bound will not fall in this range that is 2. Same logic, 2,3, 4 go to europe. 5,6 again to america. and 6.9 to austratila.

And best is from 9 to infinity. Any prefix from 9 to anythong will go to europe.

Now will enable sharding and shared messges collection

We see 1 chunk already created.

Once you enable sharding , you would like to move the chunks to there repective shards. and that would the right direction.
I will ask mongo to move the chunk with prefix 0 to america shard. ALso prefix to move to america and so on for europe chunks.

Lets see sh.status, see chunks moved as per the tags

Chunks has the min key and max key.

Friday, September 6, 2019

MongoDB : Write concerns

Durability is that the data is written to disk and that data will be there even after a server reboot. Mongodb provides various durability options based on your requirement.

1.) Acknowledged write concern
2.) Journaled write concern
3.) MultiMember Write concerns

1.) Acknowledged write concern : provides no durability guarantee. Acknowledged meand that mongo updates its in memory view with the write. It said nothing that data is anywhere on disk. there is also unacknowledgd write, which just write the data and raise concern only if there is syntax error.

2.) Journaled write concern: with Journaled write concern mongo will hold of returning OK untill write has made it into the journal. the journal is flushed to disk fairly often, so your maximum wait has a fairly low upper bound to begin with. Once a document is in the journal and on disk , a machine restart will be able to apply data. this is a minimal guarantee.

3.) MultiMember Write concerns: you can supply a write concerns that states one or more members have to acknowledge the write. Both a write concern of one and the acknowledge write concern; get the single member response from the primary. After all any wrtie in mongo has to go to primary first. So it the primary provides acknowkdge of one in both cased. A write concern of 2 means, that mongo will acknowlgde a write only after 2 members have acknowldged it. Any number higer that 1, mongo will wait for that many members to acknowdge the write and say OK to the application. this is a higer degree of durability.
Another convientt way to specify replication based write concern is to pass the majority flag. This flag means the acknowlgement by as may as the majority of member need to acknowlge the write.

From version 3.0 onwards, majority means , majority of voting members.

## Timeouts:
When you are using a multimember write concern, you are increasing the risk that one of the members will be slow, blocking or unavailable.
What is desired no of acknowldgement cant be achived, then what? surely , your client would wnat some failure indication, also it would be nice to specify some timeouts.
Such as if you cant get the acknowlegment in 3 seconds, return anyway. This kind of balance in the application be reponsive in the fase of slowneess or secondary memebr temporarily unavailable.
if the timeout has elapsed but the write concern has not been fulfilled, the application will recive an error indication. A timeout specific error so the application can detect the cause.
this condition dont cancel the write and does not roll back anything. All it does is let the application continue on and put a maximum bound of time tat that write will take. Provide an SLA to the applications.

Example:

1.) No Gurantee about nothing:

r1:PRIMARY> use demo
switched to db demo
r1:PRIMARY> db.demo.insert({x:'hi'},{writeConcern: {w:0}})
WriteResult({ })
r1:PRIMARY>

you can see the write didnt give an acknowlegement. if the server is re-started before this docuemtn make it to the disk, this document could be lost forever.

Now i will ask the write concern to be persisted to disk, write concern to be journaled equals true.

2.) Journaled write conerns. j is for journaled.

r1:PRIMARY> db.demo.insert({x:'hi'},{writeConcern: {j: true}})

WriteResult({ "nInserted" : 1 })

r1:PRIMARY>

it returned that 1 document is written and more important if the server is restarted the data wil be available.

3.) MultiMember Write concern: i am gonna ask for acknowlegde from 1 server.

r1:PRIMARY> db.demo.insert({x:'hifor 1'},{writeConcern: {w:1}})
WriteResult({ "nInserted" : 1 })
r1:PRIMARY> db.demo.find()
{ "_id" : ObjectId("5d6fa0dcf20b6b8747ecd220"), "x" : "hi" }
{ "_id" : ObjectId("5d6fa0f2f20b6b8747ecd221"), "x" : "hi" }
{ "_id" : ObjectId("5d6fa101f20b6b8747ecd222"), "x" : "hi" }
{ "_id" : ObjectId("5d6fa282f20b6b8747ecd223"), "x" : "hifor 1" }

now i gonna ask for 2 members to acknowledge my wrtie

r1:PRIMARY> db.demo.insert({x:'hifor 1'},{writeConcern: {w:2}})

WriteResult({ "nInserted" : 1 })

r1:PRIMARY> db.demo.find()

{ "_id" : ObjectId("5d6fa0dcf20b6b8747ecd220"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa0f2f20b6b8747ecd221"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa101f20b6b8747ecd222"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa282f20b6b8747ecd223"), "x" : "hifor 1" }

{ "_id" : ObjectId("5d6fa295f20b6b8747ecd224"), "x" : "hifor 1" }

r1:PRIMARY>

Here the "j" parameter is for journaled and "w" for write. Journaled and write concerns of multiple servers are not mutually exclusive. So i can say that, i want mongo to only acknowledge after the write has made it to the journal and two members have acknowledged.

r1:PRIMARY> db.demo.insert({x:'hifor 1'},{writeConcern: {w:2, j:true}})

WriteResult({ "nInserted" : 1 })

r1:PRIMARY> db.demo.find()

{ "_id" : ObjectId("5d6fa0dcf20b6b8747ecd220"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa0f2f20b6b8747ecd221"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa101f20b6b8747ecd222"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa282f20b6b8747ecd223"), "x" : "hifor 1" }

{ "_id" : ObjectId("5d6fa295f20b6b8747ecd224"), "x" : "hifor 1" }

{ "_id" : ObjectId("5d6fa397f20b6b8747ecd225"), "x" : "hifor 1" }

r1:PRIMARY>

Now if i want to get acknowledged my the majority field of the replica set then

r1:PRIMARY> db.demo.insert({x:'hi for majority'},{writeConcern: {w:'majority'}})

WriteResult({ "nInserted" : 1 })

r1:PRIMARY> db.demo.find()

{ "_id" : ObjectId("5d6fa0dcf20b6b8747ecd220"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa0f2f20b6b8747ecd221"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa101f20b6b8747ecd222"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa282f20b6b8747ecd223"), "x" : "hifor 1" }

{ "_id" : ObjectId("5d6fa295f20b6b8747ecd224"), "x" : "hifor 1" }

{ "_id" : ObjectId("5d6fa397f20b6b8747ecd225"), "x" : "hifor 1" }

{ "_id" : ObjectId("5d6fa3e1f20b6b8747ecd226"), "x" : "hi for majority" }

r1:PRIMARY>

_____________________________________________________________

Example for timeout: Lets kill one member and then ask for acknowledgemtn from 3 members.

[mongod@sikki4u2c ~]$ ps -ef| grep mongod

mongod 7486 1 0 11:30 ? 00:00:05 mongod --dbpath /home/mongod/m1 --port 30001 --replSet r1 --logpath /var/log/mongodb/mongod_m1.log --fork

mongod 7518 1 0 11:31 ? 00:00:04 mongod --dbpath /home/mongod/m2 --port 30002 --replSet r1 --logpath /var/log/mongodb/mongod_m2.log --fork

mongod 7547 1 0 11:31 ? 00:00:04 mongod --dbpath /home/mongod/m3 --port 30003 --replSet r1 --logpath /var/log/mongodb/mongod_m3.log --fork

[mongod@sikki4u2c ~]$ kill 7547

I am now with 2 members alive and write concerns for 3 member with timeout of 2 seconds

r1:PRIMARY> db.demo.insert({x:'hi for timeout'},{writeConcern: {w:3, wtimeout:2000}})

WriteResult({

"nInserted" : 1, -- it inserted even after a timeout

"writeConcernError" : {

"code" : 64,

"codeName" : "WriteConcernFailed",

"errInfo" : {

"wtimeout" : true --- timeout was true

"errmsg" : "waiting for replication timed out"

}

})

r1:PRIMARY>

this error does not mean that the document wasnt persisted anywhere, it just means the document didnt get acknowledgement from 3 members before the timeout. And the docuemnt may have made it to the secondary.

r1:PRIMARY> db.demo.find()

{ "_id" : ObjectId("5d6fa0dcf20b6b8747ecd220"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa0f2f20b6b8747ecd221"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa101f20b6b8747ecd222"), "x" : "hi" }

{ "_id" : ObjectId("5d6fa282f20b6b8747ecd223"), "x" : "hifor 1" }

{ "_id" : ObjectId("5d6fa295f20b6b8747ecd224"), "x" : "hifor 1" }

{ "_id" : ObjectId("5d6fa397f20b6b8747ecd225"), "x" : "hifor 1" }

{ "_id" : ObjectId("5d6fa3e1f20b6b8747ecd226"), "x" : "hi for majority" }

{ "_id" : ObjectId("5d6fa4bff20b6b8747ecd227"), "x" : "hi for timeout" }

Tuesday, September 3, 2019

MongoDB : 3 member replica set configuration and few scenarios

mkdir m1 m2 m3

mongod --dbpath /home/mongod/m1 --port 30001 --replSet r1 --logpath /var/log/mongodb/mongod_m1.log --fork
mongod --dbpath /home/mongod/m2 --port 30002 --replSet r1 --logpath /var/log/mongodb/mongod_m2.log --fork

mongod --dbpath /home/mongod/m3 --port 30003 --replSet r1 --logpath /var/log/mongodb/mongod_m3.log --fork

[mongod@sikki4u1c ~]$ ps -ef| grep mongod

root 2518 2421 0 10:50 pts/0 00:00:00 su - mongod

mongod 2547 2518 0 10:50 pts/0 00:00:00 -bash

mongod 2644 1 5 10:50 ? 00:00:00 mongod --dbpath /home/mongod/m1 --port 30001 --replSet r1 --logpath /var/log/mongodb/mongod_m1.log --fork

mongod 2677 1 9 10:51 ? 00:00:00 mongod --dbpath /home/mongod/m2 --port 30002 --replSet r1 --logpath /var/log/mongodb/mongod_m2.log --fork

mongod 2712 1 15 10:51 ? 00:00:00 mongod --dbpath /home/mongod/m3 --port 30003 --replSet r1 --logpath /var/log/mongodb/mongod_m3.log --fork

mongod 2743 2547 0 10:51 pts/0 00:00:00 ps -ef

mongod 2744 2547 0 10:51 pts/0 00:00:00 grep --color=auto mongod

[mongod@sikki4u1c ~]$

I have started 3 mongod instance on 3 different data directory and port.

Now lets connect to any one instance and do rs.initiate

mongo --port 30001

MongoDB shell version v3.6.13

> rs.initiate()

{

"info2" : "no configuration specified. Using a default configuration for the set",

"me" : "localhost:30001",

"ok" : 1,

"operationTime" : Timestamp(1566816743, 1),

"$clusterTime" : {

"clusterTime" : Timestamp(1566816743, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

r1:SECONDARY> --- press enter

r1:SECONDARY> ---- press enter

r1:PRIMARY>

Now instance running on port 30001 in my primary. Now let add member to replicaset.

r1:PRIMARY> rs.add("localhost:30002")

{

"ok" : 1,

"operationTime" : Timestamp(1566816857, 1),

"$clusterTime" : {

"clusterTime" : Timestamp(1566816857, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

Adding 3rd node a arbiter----

r1:PRIMARY> rs.add("localhost:30003", true)

{

"ok" : 1,

"operationTime" : Timestamp(1566817075, 1),

"$clusterTime" : {

"clusterTime" : Timestamp(1566817075, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

r1:PRIMARY> rs.config()

{

"_id" : "r1",

"version" : 5,

"protocolVersion" : NumberLong(1),

"members" : [

{

"_id" : 0,

"host" : "localhost:30001",

"arbiterOnly" : false,

"buildIndexes" : true,

"hidden" : false,

"priority" : 1,

"tags" : {

"slaveDelay" : NumberLong(0),

"votes" : 1

{

"_id" : 1,

"host" : "localhost:30002",

"arbiterOnly" : false,

"buildIndexes" : true,

"hidden" : false,

"priority" : 1,

"tags" : {

"slaveDelay" : NumberLong(0),

"votes" : 1

{

"_id" : 2,

"host" : "localhost:30003",

"arbiterOnly" : true,

"buildIndexes" : true,

"hidden" : false,

"priority" : 0,

"tags" : {

"slaveDelay" : NumberLong(0),

"votes" : 1

}

"settings" : {

"chainingAllowed" : true,

"heartbeatIntervalMillis" : 2000,

"heartbeatTimeoutSecs" : 10,

"electionTimeoutMillis" : 10000,

"catchUpTimeoutMillis" : -1,

"catchUpTakeoverDelayMillis" : 30000,

"getLastErrorModes" : {

"getLastErrorDefaults" : {

"w" : 1,

"wtimeout" : 0

"replicaSetId" : ObjectId("5d63b9e7c542e89fcc1cd25e")

}

r1:PRIMARY>

r1:PRIMARY> rs.status()

{

"set" : "r1",

"date" : ISODate("2019-08-26T10:59:13.580Z"),

"myState" : 1,

"term" : NumberLong(1),

"syncingTo" : "",

"syncSourceHost" : "",

"syncSourceId" : -1,

"heartbeatIntervalMillis" : NumberLong(2000),

"optimes" : {

"lastCommittedOpTime" : {

"ts" : Timestamp(1566817145, 1),

"t" : NumberLong(1)

"readConcernMajorityOpTime" : {

"ts" : Timestamp(1566817145, 1),

"t" : NumberLong(1)

"appliedOpTime" : {

"ts" : Timestamp(1566817145, 1),

"t" : NumberLong(1)

"durableOpTime" : {

"ts" : Timestamp(1566817145, 1),

"t" : NumberLong(1)

}

"members" : [

{

"_id" : 0,

"name" : "localhost:30001",

"health" : 1,

"state" : 1,

"stateStr" : "PRIMARY",

"uptime" : 499,

"optime" : {

"ts" : Timestamp(1566817145, 1),

"t" : NumberLong(1)

"optimeDate" : ISODate("2019-08-26T10:59:05Z"),

"syncingTo" : "",

"syncSourceHost" : "",

"syncSourceId" : -1,

"infoMessage" : "",

"electionTime" : Timestamp(1566816743, 2),

"electionDate" : ISODate("2019-08-26T10:52:23Z"),

"configVersion" : 5,

"self" : true,

"lastHeartbeatMessage" : ""

{

"_id" : 1,

"name" : "localhost:30002",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 296,

"optime" : {

"ts" : Timestamp(1566817145, 1),

"t" : NumberLong(1)

"optimeDurable" : {

"ts" : Timestamp(1566817145, 1),

"t" : NumberLong(1)

"optimeDate" : ISODate("2019-08-26T10:59:05Z"),

"optimeDurableDate" : ISODate("2019-08-26T10:59:05Z"),

"lastHeartbeat" : ISODate("2019-08-26T10:59:11.819Z"),

"lastHeartbeatRecv" : ISODate("2019-08-26T10:59:11.819Z"),

"pingMs" : NumberLong(0),

"lastHeartbeatMessage" : "",

"syncingTo" : "localhost:30001",

"syncSourceHost" : "localhost:30001",

"syncSourceId" : 0,

"infoMessage" : "",

"configVersion" : 5

{

"_id" : 2,

"name" : "localhost:30003",

"health" : 1,

"state" : 7,

"stateStr" : "ARBITER",

"uptime" : 13,

"lastHeartbeat" : ISODate("2019-08-26T10:59:11.848Z"),

"lastHeartbeatRecv" : ISODate("2019-08-26T10:59:11.644Z"),

"pingMs" : NumberLong(0),

"lastHeartbeatMessage" : "",

"syncingTo" : "",

"syncSourceHost" : "",

"syncSourceId" : -1,

"infoMessage" : "",

"configVersion" : 5

}

"ok" : 1,

"operationTime" : Timestamp(1566817145, 1),

"$clusterTime" : {

"clusterTime" : Timestamp(1566817145, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

_______ I see all 3 nodes added to replicaset -----

Lets insert some dummy data using a java script. I am on primary

r1:PRIMARY> for(i=0; i<10; i++){db.demo.save({_id:i});}

WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : 9 })

r1:PRIMARY> db.demo.find()

{ "_id" : 0 }

{ "_id" : 1 }

{ "_id" : 2 }

{ "_id" : 3 }

{ "_id" : 4 }

{ "_id" : 5 }

{ "_id" : 6 }

{ "_id" : 7 }

{ "_id" : 8 }

{ "_id" : 9 }

r1:PRIMARY>

let connect to secondary.

r1:PRIMARY> db = connect("localhost:30002/test")

connecting to: mongodb://localhost:30002/test

Implicit session: session { "id" : UUID("6cf22711-ebf8-40ff-90fb-4f461d6064c8") }

MongoDB server version: 3.6.13

test

r1:SECONDARY> db.demo.find()

Error: error: {

"operationTime" : Timestamp(1566817515, 1),

"ok" : 0,

"errmsg" : "not master and slaveOk=false",

"code" : 13435,

"codeName" : "NotMasterNoSlaveOk",

"$clusterTime" : {

"clusterTime" : Timestamp(1566817515, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

r1:SECONDARY> rs.slaveOk()

r1:SECONDARY> db.demo.find()

{ "_id" : 0 }

{ "_id" : 4 }

{ "_id" : 2 }

{ "_id" : 3 }

{ "_id" : 7 }

{ "_id" : 6 }

{ "_id" : 1 }

{ "_id" : 5 }

{ "_id" : 8 }

{ "_id" : 9 }

r1:SECONDARY>

here you go.. i see the data in 2nd mongod instance.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Now i am on secondary and i will kill primary to see what happens

[mongod@sikki4u1c ~]$ ps -ef| grep mongod

root 2518 2421 0 10:50 pts/0 00:00:00 su - mongod

mongod 2547 2518 0 10:50 pts/0 00:00:00 -bash

mongod 2644 1 0 10:50 ? 00:00:05 mongod --dbpath /home/mongod/m1 --port 30001 --replSet r1 --logpath /var/log/mongodb/mongod_m1.log --fork

mongod 2677 1 0 10:51 ? 00:00:05 mongod --dbpath /home/mongod/m2 --port 30002 --replSet r1 --logpath /var/log/mongodb/mongod_m2.log --fork

mongod 3132 1 0 10:58 ? 00:00:02 mongod --dbpath /home/mongod/m3 --port 30003 --replSet r1 --logpath /var/log/mongodb/mongod_m3.log --fork

mongod 3302 2547 0 11:04 pts/0 00:00:00 mongo --port 30001

root 3489 3395 0 11:07 pts/1 00:00:00 su - mongod

mongod 3493 3489 0 11:07 pts/1 00:00:00 -bash

mongod 3587 3493 0 11:07 pts/1 00:00:00 ps -ef

mongod 3588 3493 0 11:07 pts/1 00:00:00 grep --color=auto mongod

[mongod@sikki4u1c ~]$ kill 2644

[mongod@sikki4u1c ~]$

Now check the replicaset status on secondary, and note the prompt changing from SECONDARY to PRIMARY. Since 2 of our 3 members were active the instance running on port 30002 was elected as primary.

r1:SECONDARY>

r1:PRIMARY> rs.status()

{

"set" : "r1",

"date" : ISODate("2019-08-26T11:08:22.387Z"),

"myState" : 1,

"term" : NumberLong(2),

"syncingTo" : "",

"syncSourceHost" : "",

"syncSourceId" : -1,

"heartbeatIntervalMillis" : NumberLong(2000),

"optimes" : {

"lastCommittedOpTime" : {

"ts" : Timestamp(1566817675, 1),

"t" : NumberLong(1)

"readConcernMajorityOpTime" : {

"ts" : Timestamp(1566817675, 1),

"t" : NumberLong(1)

"appliedOpTime" : {

"ts" : Timestamp(1566817693, 1),

"t" : NumberLong(2)

"durableOpTime" : {

"ts" : Timestamp(1566817693, 1),

"t" : NumberLong(2)

}

"members" : [

{

"_id" : 0,

"name" : "localhost:30001",

"health" : 0,

"state" : 8,

"stateStr" : "(not reachable/healthy)",

"uptime" : 0,

"optime" : {

"ts" : Timestamp(0, 0),

"t" : NumberLong(-1)

"optimeDurable" : {

"ts" : Timestamp(0, 0),

"t" : NumberLong(-1)

"optimeDate" : ISODate("1970-01-01T00:00:00Z"),

"optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),

"lastHeartbeat" : ISODate("2019-08-26T11:08:22.338Z"),

"lastHeartbeatRecv" : ISODate("2019-08-26T11:07:59.918Z"),

"pingMs" : NumberLong(0),

"lastHeartbeatMessage" : "Connection refused",

"syncingTo" : "",

"syncSourceHost" : "",

"syncSourceId" : -1,

"infoMessage" : "",

"configVersion" : -1

{

"_id" : 1,

"name" : "localhost:30002",

"health" : 1,

"state" : 1,

"stateStr" : "PRIMARY",

"uptime" : 1042,

"optime" : {

"ts" : Timestamp(1566817693, 1),

"t" : NumberLong(2)

"optimeDate" : ISODate("2019-08-26T11:08:13Z"),

"syncingTo" : "",

"syncSourceHost" : "",

"syncSourceId" : -1,

"infoMessage" : "could not find member to sync from",

"electionTime" : Timestamp(1566817692, 1),

"electionDate" : ISODate("2019-08-26T11:08:12Z"),

"configVersion" : 5,

"self" : true,

"lastHeartbeatMessage" : ""

{

"_id" : 2,

"name" : "localhost:30003",

"health" : 1,

"state" : 7,

"stateStr" : "ARBITER",

"uptime" : 562,

"lastHeartbeat" : ISODate("2019-08-26T11:08:22.333Z"),

"lastHeartbeatRecv" : ISODate("2019-08-26T11:08:21.773Z"),

"pingMs" : NumberLong(0),

"lastHeartbeatMessage" : "",

"syncingTo" : "",

"syncSourceHost" : "",

"syncSourceId" : -1,

"infoMessage" : "",

"configVersion" : 5

}

"ok" : 1,

"operationTime" : Timestamp(1566817693, 1),

"$clusterTime" : {

"clusterTime" : Timestamp(1566817693, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

r1:PRIMARY>

_______________________________________

Now lets try to kill the arbiter and check the status.

mongod 2677 1 0 10:51 ? 00:00:06 mongod --dbpath /home/mongod/m2 --port 30002 --replSet r1 --logpath /var/log/mongodb/mongod_m2.log --fork

mongod 3132 1 0 10:58 ? 00:00:03 mongod --dbpath /home/mongod/m3 --port 30003 --replSet r1 --logpath /var/log/mongodb/mongod_m3.log --fork

mongod 3302 2547 0 11:04 pts/0 00:00:00 mongo --port 30001

root 3489 3395 0 11:07 pts/1 00:00:00 su - mongod

mongod 3493 3489 0 11:07 pts/1 00:00:00 -bash

mongod 3687 3493 0 11:11 pts/1 00:00:00 ps -ef

mongod 3688 3493 0 11:11 pts/1 00:00:00 grep --color=auto mongod

[mongod@sikki4u1c ~]$ kill 3132

r1:PRIMARY> rs.status()

2019-08-26T11:11:42.702+0000 E QUERY [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetGetStatus' on host 'localhost:30002' :

DB.prototype.runCommand@src/mongo/shell/db.js:168:1

DB.prototype.adminCommand@src/mongo/shell/db.js:186:16

rs.status@src/mongo/shell/utils.js:1268:12

@(shell):1:1

2019-08-26T11:11:42.704+0000 I NETWORK [thread1] trying reconnect to localhost:30002 (127.0.0.1) failed

2019-08-26T11:11:42.704+0000 I NETWORK [thread1] reconnect localhost:30002 (127.0.0.1) ok

r1:SECONDARY>

Mongo has shut down the connections server to prevent and writing to primary. this mechanism prevents application from trying to write to the previous primary.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Priority:

Here i will set priority for member [0] to 10. as of now all are having same priority.

r1:PRIMARY> rs.config()
{
"_id" : "r1",
"version" : 5,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 0,
"host" : "localhost:30001",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 1,
"host" : "localhost:30002",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 2,
"host" : "localhost:30003",
"arbiterOnly" : true,
"buildIndexes" : true,
"hidden" : false,
"priority" : 0,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"catchUpTimeoutMillis" : -1,
"catchUpTakeoverDelayMillis" : 30000,
"getLastErrorModes" : {

},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("5d63b9e7c542e89fcc1cd25e")
}
}
r1:PRIMARY> var cfg = rs.config()
r1:PRIMARY> cfg.members[0].priority = 10
10
r1:PRIMARY> rs.reconfig(cfg)
{
"ok" : 1,
"operationTime" : Timestamp(1567060050, 2),
"$clusterTime" : {
"clusterTime" : Timestamp(1567060050, 2),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
r1:PRIMARY> rs.config()
{
"_id" : "r1",
"version" : 6,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 0,
"host" : "localhost:30001",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 10,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 1,
"host" : "localhost:30002",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 2,
"host" : "localhost:30003",
"arbiterOnly" : true,
"buildIndexes" : true,
"hidden" : false,
"priority" : 0,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"catchUpTimeoutMillis" : -1,
"catchUpTakeoverDelayMillis" : 30000,
"getLastErrorModes" : {

},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("5d63b9e7c542e89fcc1cd25e")
}
}
r1:PRIMARY>

How do i now which mongo server i am conncted to

r1:PRIMARY> db.getMongo()
connection to 127.0.0.1:30001
r1:PRIMARY> db=connect('127.0.0.1:30002/test')
connecting to: mongodb://127.0.0.1:30002/test
Implicit session: session { "id" : UUID("34172119-afd2-4a8c-862a-f9789f2878c3") }
MongoDB server version: 3.6.13
test
r1:SECONDARY>

---here i connected to secondary and now i will kill my primary

as i kill primary the second instance running on port 30002 becomes primary.

Now again if i start my first mongod instance on 30001, it will become primary again as it have higher priority over other nodes.

++++++++++++++++++++++++++++++++++++++++++++++++++++

Freeze: function is used to stop a secondary node from taking a primary role for a mentioned time period

rs.freeze(5*60)
i am asking secondary to freeze for 60*5 = 300 Secs.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Hidden node: application cannot see the hidden member it can be used for reporting or other work load, without interfering wtih the application load . hidden member cant become primatry ever. But it can vote in election.

var cfg = rs.config()
cfg.members[2].priority=0 --- 0 priority means, it will never become primary
cfg.members[2].hidden=true -- make it hidden
rs.recofig(cfg)

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Mongo DB Journal

Saturday, September 14, 2019

MongoDB: Security

Friday, September 13, 2019

MOngoDB : Monitoring

Tuesday, September 10, 2019

MongoDB: Sharding Concepts

MOngoDb : Sharding Keys

Friday, September 6, 2019

MongoDB : Write concerns

Tuesday, September 3, 2019

MongoDB : 3 member replica set configuration and few scenarios

Links

Previous Posts

Archives