Metrics
1. Cassandra outputs an overview of the cluster by using the following
Commands
nodetool status
nodetool status <keyspace-name>
Output
admatic@admatic-VirtualBox:/var/lib/cassandra/data/admatic/emp-20e70b2030d511e8bdf32b06aa808981$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 127.0.0.1 298.61 KiB 256 100.0% 6e3e835b-bee3-4b86-9035-b4d2921c1ae7 rack1
2. 'nodetool info' outputs slightly more detailed statistics for an individual node in the cluster, including uptime, load, key cache hit rate, and a total count of all exceptions
Command
nodetool info
Output
admatic@admatic-VirtualBox:/var/lib/cassandra/data/admatic/emp-20e70b2030d511e8bdf32b06aa808981$ nodetool --host 127.0.0.1 info
ID : 6e3e835b-bee3-4b86-9035-b4d2921c1ae7
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 298.61 KiB
Generation No : 1522054227
Uptime (seconds) : 11345
Heap Memory (MB) : 89.96 / 1014.00
Off Heap Memory (MB) : 0.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : entries 19, size 1.43 KiB, capacity 50 MiB, 178 hits, 217 requests, 0.820 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 0, size 0 bytes, capacity 25 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache : entries 18, size 1.12 MiB, capacity 221 MiB, 56 misses, 450 requests, 0.876 recent hit rate, NaN microseconds miss latency
Percent Repaired : 0.0%
Token : (invoke with -T/--tokens to see all 256 tokens)
admatic@admatic-VirtualBox:/var/lib/cassandra/data/admatic/emp-20e70b2030d511e8bdf32b06aa808981$
3. 'nodetool cfstats' provides statistics on each keyspace and column family (akin to databases and database tables, respectively), including read latency, write latency, and total disk space used
Command
nodetool cfstats demo
Output
admatic@admatic-VirtualBox:/var/lib/cassandra/data/admatic/emp-20e70b2030d511e8bdf32b06aa808981$ nodetool cfstats
Total number of tables: 37
----------------
Keyspace : system_traces
Read Count: 0
Read Latency: NaN ms
Write Count: 0
Write Latency: NaN ms
Pending Flushes: 0
Table: events
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: sessions
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
----------------
Keyspace : system
Read Count: 49
Read Latency: 6.458836734693878 ms
Write Count: 530
Write Latency: 1.522566037735849 ms
Pending Flushes: 0
Table: IndexInfo
SSTable count: 2
Space used (live): 9894
Space used (total): 9894
Space used by snapshots (total): 9894
Off heap memory used (total): 70
SSTable Compression Ratio: 1.208955223880597
Number of partitions (estimate): 1
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 2
Local read count: 2
Local read latency: NaN ms
Local write count: 2
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 32
Bloom filter off heap memory used: 16
Index summary off heap memory used: 38
Compression metadata off heap memory used: 16
Compacted partition minimum bytes: 30
Compacted partition maximum bytes: 35
Compacted partition mean bytes: 35
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: available_ranges
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: batches
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 1
Local read count: 0
Local read latency: NaN ms
Local write count: 2
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1
Dropped Mutations: 0
Table: batchlog
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: built_views
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: compaction_history
SSTable count: 3
Space used (live): 16635
Space used (total): 16635
Space used by snapshots (total): 11257
Off heap memory used (total): 140
SSTable Compression Ratio: 0.6630076838638859
Number of partitions (estimate): 8
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 3
Local read count: 0
Local read latency: NaN ms
Local write count: 8
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 56
Bloom filter off heap memory used: 32
Index summary off heap memory used: 84
Compression metadata off heap memory used: 24
Compacted partition minimum bytes: 87
Compacted partition maximum bytes: 149
Compacted partition mean bytes: 122
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: hints
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: local
SSTable count: 2
Space used (live): 16021
Space used (total): 16021
Space used by snapshots (total): 25801
Off heap memory used (total): 74
SSTable Compression Ratio: 0.767544504928645
Number of partitions (estimate): 1
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 14
Local read count: 40
Local read latency: NaN ms
Local write count: 25
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 32
Bloom filter off heap memory used: 16
Index summary off heap memory used: 34
Compression metadata off heap memory used: 24
Compacted partition minimum bytes: 61
Compacted partition maximum bytes: 6866
Compacted partition mean bytes: 3469
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: paxos
SSTable count: 0
SSTables in each level: [0, 0, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: peer_events
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: peers
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 1
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: prepared_statements
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: range_xfers
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: size_estimates
SSTable count: 3
Space used (live): 143181
Space used (total): 143181
Space used by snapshots (total): 95472
Off heap memory used (total): 189
SSTable Compression Ratio: 0.19875132205078253
Number of partitions (estimate): 8
Memtable cell count: 15480
Memtable data size: 311066
Memtable off heap memory used: 0
Memtable switch count: 3
Local read count: 0
Local read latency: NaN ms
Local write count: 386
Local write latency: 15.932 ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 48
Bloom filter off heap memory used: 24
Index summary off heap memory used: 69
Compression metadata off heap memory used: 96
Compacted partition minimum bytes: 17085
Compacted partition maximum bytes: 88148
Compacted partition mean bytes: 56154
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: sstable_activity
SSTable count: 3
Space used (live): 17977
Space used (total): 17977
Space used by snapshots (total): 12323
Off heap memory used (total): 209
SSTable Compression Ratio: 0.44407265774378585
Number of partitions (estimate): 44
Memtable cell count: 17
Memtable data size: 255
Memtable off heap memory used: 0
Memtable switch count: 3
Local read count: 6
Local read latency: NaN ms
Local write count: 107
Local write latency: 0.032 ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 96
Bloom filter off heap memory used: 72
Index summary off heap memory used: 113
Compression metadata off heap memory used: 24
Compacted partition minimum bytes: 36
Compacted partition maximum bytes: 86
Compacted partition mean bytes: 50
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: transferred_ranges
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: views_builds_in_progress
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
----------------
Keyspace : system_distributed
Read Count: 0
Read Latency: NaN ms
Write Count: 0
Write Latency: NaN ms
Pending Flushes: 0
Table: parent_repair_history
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: repair_history
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: view_build_status
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
----------------
Keyspace : system_schema
Read Count: 179
Read Latency: 1.1957653631284915 ms
Write Count: 49
Write Latency: 1.964061224489796 ms
Pending Flushes: 0
Table: aggregates
SSTable count: 1
Space used (live): 5065
Space used (total): 5065
Space used by snapshots (total): 5065
Off heap memory used (total): 41
SSTable Compression Ratio: 1.0408163265306123
Number of partitions (estimate): 2
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 1
Local read count: 9
Local read latency: NaN ms
Local write count: 2
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 16
Bloom filter off heap memory used: 8
Index summary off heap memory used: 25
Compression metadata off heap memory used: 8
Compacted partition minimum bytes: 21
Compacted partition maximum bytes: 29
Compacted partition mean bytes: 27
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: columns
SSTable count: 2
Space used (live): 16419
Space used (total): 16419
Space used by snapshots (total): 16419
Off heap memory used (total): 90
SSTable Compression Ratio: 0.3260170712781288
Number of partitions (estimate): 6
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 5
Local read count: 22
Local read latency: NaN ms
Local write count: 9
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 40
Bloom filter off heap memory used: 24
Index summary off heap memory used: 42
Compression metadata off heap memory used: 24
Compacted partition minimum bytes: 73
Compacted partition maximum bytes: 8239
Compacted partition mean bytes: 2816
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: dropped_columns
SSTable count: 1
Space used (live): 4979
Space used (total): 4979
Space used by snapshots (total): 4979
Off heap memory used (total): 41
SSTable Compression Ratio: 1.0408163265306123
Number of partitions (estimate): 2
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 1
Local read count: 14
Local read latency: NaN ms
Local write count: 2
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 16
Bloom filter off heap memory used: 8
Index summary off heap memory used: 25
Compression metadata off heap memory used: 8
Compacted partition minimum bytes: 21
Compacted partition maximum bytes: 29
Compacted partition mean bytes: 27
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: functions
SSTable count: 1
Space used (live): 5065
Space used (total): 5065
Space used by snapshots (total): 5065
Off heap memory used (total): 41
SSTable Compression Ratio: 1.0408163265306123
Number of partitions (estimate): 2
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 1
Local read count: 9
Local read latency: NaN ms
Local write count: 2
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 16
Bloom filter off heap memory used: 8
Index summary off heap memory used: 25
Compression metadata off heap memory used: 8
Compacted partition minimum bytes: 21
Compacted partition maximum bytes: 29
Compacted partition mean bytes: 27
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: indexes
SSTable count: 3
Space used (live): 15149
Space used (total): 15149
Space used by snapshots (total): 15149
Off heap memory used (total): 111
SSTable Compression Ratio: 1.1111111111111112
Number of partitions (estimate): 3
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 3
Local read count: 22
Local read latency: NaN ms
Local write count: 4
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 48
Bloom filter off heap memory used: 24
Index summary off heap memory used: 63
Compression metadata off heap memory used: 24
Compacted partition minimum bytes: 21
Compacted partition maximum bytes: 86
Compacted partition mean bytes: 45
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: keyspaces
SSTable count: 3
Space used (live): 15793
Space used (total): 15793
Space used by snapshots (total): 15793
Off heap memory used (total): 125
SSTable Compression Ratio: 0.564185544768069
Number of partitions (estimate): 6
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 9
Local read count: 21
Local read latency: NaN ms
Local write count: 13
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 56
Bloom filter off heap memory used: 32
Index summary off heap memory used: 61
Compression metadata off heap memory used: 32
Compacted partition minimum bytes: 87
Compacted partition maximum bytes: 149
Compacted partition mean bytes: 122
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: tables
SSTable count: 1
Space used (live): 9400
Space used (total): 9400
Space used by snapshots (total): 9400
Off heap memory used (total): 55
SSTable Compression Ratio: 0.16974012537277097
Number of partitions (estimate): 6
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 7
Local read count: 34
Local read latency: NaN ms
Local write count: 11
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 24
Bloom filter off heap memory used: 16
Index summary off heap memory used: 23
Compression metadata off heap memory used: 16
Compacted partition minimum bytes: 373
Compacted partition maximum bytes: 8239
Compacted partition mean bytes: 2982
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: triggers
SSTable count: 1
Space used (live): 4979
Space used (total): 4979
Space used by snapshots (total): 4979
Off heap memory used (total): 41
SSTable Compression Ratio: 1.0408163265306123
Number of partitions (estimate): 2
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 1
Local read count: 22
Local read latency: NaN ms
Local write count: 2
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 16
Bloom filter off heap memory used: 8
Index summary off heap memory used: 25
Compression metadata off heap memory used: 8
Compacted partition minimum bytes: 21
Compacted partition maximum bytes: 29
Compacted partition mean bytes: 27
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: types
SSTable count: 1
Space used (live): 4938
Space used (total): 4938
Space used by snapshots (total): 4938
Off heap memory used (total): 41
SSTable Compression Ratio: 1.0408163265306123
Number of partitions (estimate): 2
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 1
Local read count: 9
Local read latency: NaN ms
Local write count: 2
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 16
Bloom filter off heap memory used: 8
Index summary off heap memory used: 25
Compression metadata off heap memory used: 8
Compacted partition minimum bytes: 21
Compacted partition maximum bytes: 29
Compacted partition mean bytes: 27
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: views
SSTable count: 1
Space used (live): 4938
Space used (total): 4938
Space used by snapshots (total): 4938
Off heap memory used (total): 41
SSTable Compression Ratio: 1.0408163265306123
Number of partitions (estimate): 2
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 1
Local read count: 17
Local read latency: NaN ms
Local write count: 2
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 16
Bloom filter off heap memory used: 8
Index summary off heap memory used: 25
Compression metadata off heap memory used: 8
Compacted partition minimum bytes: 21
Compacted partition maximum bytes: 29
Compacted partition mean bytes: 27
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
----------------
Keyspace : admatic
Read Count: 5
Read Latency: 4.1788 ms
Write Count: 9
Write Latency: 0.09988888888888889 ms
Pending Flushes: 0
Table: emp
SSTable count: 2
Space used (live): 10269
Space used (total): 10269
Space used by snapshots (total): 41076
Off heap memory used (total): 64
SSTable Compression Ratio: 0.8553191489361702
Number of partitions (estimate): 4
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 4
Local read count: 5
Local read latency: NaN ms
Local write count: 9
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 32
Bloom filter off heap memory used: 16
Index summary off heap memory used: 32
Compression metadata off heap memory used: 16
Compacted partition minimum bytes: 18
Compacted partition maximum bytes: 72
Compacted partition mean bytes: 43
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
----------------
Keyspace : system_auth
Read Count: 2
Read Latency: 0.424 ms
Write Count: 1
Write Latency: 0.195 ms
Pending Flushes: 0
Table: resource_role_permissons_index
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: role_members
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: role_permissions
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: -1.0
Number of partitions (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Table: roles
SSTable count: 1
Space used (live): 5070
Space used (total): 5070
Space used by snapshots (total): 5070
Off heap memory used (total): 37
SSTable Compression Ratio: 1.0851063829787233
Number of partitions (estimate): 1
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 1
Local read count: 2
Local read latency: NaN ms
Local write count: 1
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 16
Bloom filter off heap memory used: 8
Index summary off heap memory used: 21
Compression metadata off heap memory used: 8
Compacted partition minimum bytes: 87
Compacted partition maximum bytes: 103
Compacted partition mean bytes: 103
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
----------------
admatic@admatic-VirtualBox:/var/lib/cassandra/data/admatic/emp-20e70b2030d511e8bdf32b06aa808981$
5. 'nodetool compactionstats' shows the compactions in progess as well as a count of pending compaction tasks
Command Run
sudo nodetool compact
On another instance of the node run
sudo nodetool compactionstats
Output
hadoop@ip-172-31-88-141:~$ sudo nodetool compact
hadoop@ip-172-31-88-141:~$ sudo nodetool compactionstats
pending tasks: 1
- keyspace1.standard1: 1
id compaction type keyspace table completed total unit progress
4bad41a0-3641-11e8-87f4-fdeaf7a1f356 Compaction keyspace1 standard1 114892624 114914235 bytes 99.98%
Active compaction remaining time : 0h00m00s
6. nodetool gcstats returns statistics on garbage collections, including total number of collections and elapsed time (both the total and the max elapsed time)
Command
nodetool gcstats
Output
admatic@admatic-VirtualBox:/var/lib/cassandra/data/admatic/emp-20e70b2030d511e8bdf32b06aa808981$ nodetool gcstats
Interval (ms) Max GC Elapsed (ms)Total GC Elapsed (ms)Stdev GC Elapsed (ms) GC Reclaimed (MB) Collections Direct Memory Bytes
11695767 213 584 51 1096161712 14 -1
admatic@admatic-VirtualBox:/var/lib/cassandra/data/admatic/emp-20e70b2030d511e8bdf32b06aa808981$
7. nodetool tpstats provides usage statistics on Cassandra’s thread pool, including pending tasks as well as current and historical blocked tasks
Command
nodetool tpstats
Output
admatic@admatic-VirtualBox:/var/lib/cassandra/data/admatic/emp-20e70b2030d511e8bdf32b06aa808981$ nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 144 0 0
MiscStage 0 0 0 0 0
CompactionExecutor 0 0 7127 0 0
MutationStage 0 0 13 0 0
MemtableReclaimMemory 0 0 63 0 0
PendingRangeCalculator 0 0 1 0 0
GossipStage 0 0 0 0 0
SecondaryIndexManagement 0 0 1 0 0
HintsDispatcher 0 0 0 0 0
RequestResponseStage 0 0 1 0 0
Native-Transport-Requests 0 0 1253 0 0
ReadRepairStage 0 0 0 0 0
CounterMutationStage 0 0 0 0 0
MigrationStage 0 0 9 0 0
MemtablePostFlush 0 0 166 0 0
PerDiskMemtableFlushWriter_0 0 0 60 0 0
ValidationExecutor 0 0 0 0 0
Sampler 0 0 0 0 0
MemtableFlushWriter 0 0 62 0 0
InternalResponseStage 0 0 0 0 0
ViewMutationStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
CacheCleanupExecutor 0 0 0 0 0
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
HINT 0
MUTATION 0
COUNTER_MUTATION 0
BATCH_STORE 0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 0
admatic@admatic-VirtualBox:/var/lib/cassandra/data/admatic/emp-20e70b2030d511e8bdf32b06aa808981$
GETTING METRICS USING Jconsole-
1. Install jre
sudo apt-get install default-jre -y
2. Install jdk
sudo apt-get install default-jdk -y
3. Open jconsole as follows
jconsole
(you can get a visual idea of different metrics on jconsole)
4. nodetool repair
nodetool repair <keyspace-name> <table-name>
eg
nodetool repair admatic
nodetool repair keyspace1
Log-data
[2018-04-02 04:27:20,699] Replication factor is 1. No repair is needed for keyspace 'admat
ic'
hadoop@ip-172-31-86-204:~$ nodetool repair keyspace1
[2018-04-02 04:27:31,747] Repair session 28bb7710-362e-11e8-bcb0-837d440845a6 for range [(
-7870482679188171803,-7818210294615788870], (-1681474227339127365,-1665455506675008751], (
6812507913924172795,6823794479633212960], (7412472837895658066,7457263962936588567], (-248
9013957256393284,-2473205192229841781], (-7216646923564598821,-7172708065983408005], (5136
957068384496641,5138485038713736537], (-2035909268737880643,-1980531580649356981]
[2018-04-02 04:27:34,703] Repair completed successfully
[2018-04-02 04:27:34,705] Repair command #3 finished in 3 seconds
hadoop@ip-172-31-86-204:~$ nodetool netstats
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name Active Pending Completed Dropped
Large messages n/a 1 3 5
Small messages n/a 0 7081 5
Gossip messages n/a 4 208312 10701
hadoop@ip-172-31-86-204:~$ nodetool netstats
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name Active Pending Completed Dropped
Large messages n/a 1 3 5
Small messages n/a 0 7081 5
Gossip messages n/a 2 208346 10706
hadoop@ip-172-31-86-204:~$
anti-entropy-repairs
Anti-entropy repair is triggered manually
often recommended to be run periodically to keep data in sync to repair fully
1. nodetool rebuild
nodetool rebuild –keyspace cyclists, motorists
2. nodetool rebuild_index
nodetool rebuild_index admatic emp name
5.0 nodetool bootstrap (will get us the bootstrapping information)
5.1 nodetool decommission (will remove healthy and running nodes only) on the node you want to leave from the cluster
nodetool decommission
Output
hadoop@ip-172-31-84-180:~$ sudo nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UL 172.31.91.105 134.27 KiB 256 66.1% 26bcba5d-255a-420b-ba5a-ca19c995011a rack1
UN 172.31.84.180 134.02 KiB 256 65.5% a9629fdc-ecc8-49c9-8759-77511bd8d569 rack1
UN 172.31.90.211 134.57 KiB 256 68.3% 617cbee7-1ecd-4951-ab44-b2b9df70538e rack1
status UL shows that the node has been decommisioned
after a while, the nodes leaves the cluster
6. How to bring back a decommissioned node?
The straightforward way to add a decommissioned node back to a cluster is to simply remove all the old data from the node, which is stored under the following directories under /data:
commitlog
data
saved_caches
eg
- go to the given directory
cd /var/lib/cassandra/
hadoop@ip-172-31-88-19:~$ cd /var/lib/cassandra/
hadoop@ip-172-31-88-19:/var/lib/cassandra$ ls
commitlog data hints saved_caches
hadoop@ip-172-31-88-19:/var/lib/cassandra$
- remove commitlog,data and saved_caches
sudo rm -r commitlog
sudo rm -r data
sudo rm -r saved_caches
restart the node to add it to the cluster
sudo service cassandra restarttype nodetool status to get all the node's status
sudo nodetool statusdescribe the cluster as follows-
nodetool describecluster
7.ASSASSINATING A NODE (nodetool assassinate)
nodetool assassinate
(when a node doesn't leave the cluster and has DN status [Down+Normal] do this)
nodetool assassinate
(removes node on which the command is run)
to remove a specific node
nodetool assassinate -h <node-ip-address>
8.If you want a node to be up/running, but don't want it inside a cluster [place node in DN state]
- Stop the node
sudo service cassandra stop
- Edit the cassandra-env.sh file and add the following line
path
sudo vim /etc/cassandra/cassandra-env.sh
- Add the following line at the end of the file
JVM_OPTS="$JVM_OPTS -Dcassandra.join_ring=false"
- Start the node again
this will remove the node from the cluster, but will keep it running
9.To remove down/unhealthy nodes
Host ID is what we get after 'nodetool status' as the 6th column as follows
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 172.31.88.141 298.16 MiB 256 ? 8876c9e0-e14e-4ed6-8f53-e74e8d0a9498 rack1
UN 172.31.86.204 457.25 MiB 256 ? 0a4a6383-a563-49ec-a47a-679ee6754606 rack1
UN 172.31.90.24 328.96 KiB 256 ? 01c60988-3bba-4dd8-8480-5ec237494b07 rack1
UN 172.31.88.19 355.61 MiB 256 ? 3e1d4eb1-faf4-4228-8112-d0fd3113c6ed rack1
general
nodetool removenode <host-id>
eg
nodetool removenode 01c60988-3bba-4dd8-8480-5ec237494b07
Note
- 'removenode' cannot remove active and running nodes
- it is used to remove down/killed nodes only
10.Moving a node to a different rack
edit the following file
sudo vim /etc/cassandra/cassandra-rackdc.properties
edit it as follows
Nodes 1 & 2
indicate the rack and dc for this node
dc=DC1
rack=RAC1
Nodes 3 & 4
indicate the rack and dc for this node
dc=DC2
rack=RAC1
11.manage gossip info
You can get gossip information from the cluster by running the nodetool gossipinfo command, as shown here
sudo nodetool gossipinfo
- disable gossip info on a node
sudo nodetool disablegossip
- enable gossip info on a node
sudo nodetool enablegossip
LOG-DATA
hadoop@ip-172-31-88-141:~$ nodetool statusgossip
running
hadoop@ip-172-31-88-141:~$ nodetool disablegossip
hadoop@ip-172-31-88-141:~$ nodetool statusgossip
not running
12.nodetool proxyhistograms
The nodetool proxyhistograms command shows the network statistics in a cluster.
13.cassandra-stress test
cassandra-stress write n=1000000 -schema "replication(strategy=SimpleStrategy,factor=3 )" -node 172.31.90.24
14.tablehistograms examples [get read,write latencies]
general
tablehistograms <keyspace-name> <table-name>
eg
nodetool tablehistograms keyspace1 standard1
15.types of cache
- key cache
- row cache
- counter cache
- chunk cache
16.tracing on/off examples
log data
cqlsh:admatic> tracing on;
Now Tracing is enabled
cqlsh:admatic> INSERT INTO emp (emp_id, emp_city, emp_name, emp_phone, emp_sal) values( 4,'Pune','rajeev',9848022331, 30000);
Tracing session: 66616a00-3423-11e8-a13d-d18824af44c6
activity | timestamp | source | source_elapsed | client
-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------+---------------+----------------+--------------
Execute CQL3 query | 2018-03-30 14:05:28.096000 | 172.31.88.19 | 0 | 172.31.88.19
Parsing INSERT INTO emp (emp_id, emp_city, emp_name, emp_phone, emp_sal) values( 4,'Pune','rajeev',9848022331, 30000); [Native-Transport-Requests-1] | 2018-03-30 14:05:28.097000 | 172.31.88.19 | 419 | 172.31.88.19
Preparing statement [Native-Transport-Requests-1] | 2018-03-30 14:05:28.097000 | 172.31.88.19 | 736 | 172.31.88.19
MUTATION message received from /172.31.88.19 [MessagingService-Incoming-/172.31.88.19] | 2018-03-30 14:05:28.103000 | 172.31.86.204 | 6 | 172.31.88.19
Determining replicas for mutation [Native-Transport-Requests-1] | 2018-03-30 14:05:28.103000 | 172.31.88.19 | 6272 | 172.31.88.19
Appending to commitlog [MutationStage-1] | 2018-03-30 14:05:28.103000 | 172.31.86.204 | 399 | 172.31.88.19
Sending MUTATION message to /172.31.86.204 [MessagingService-Outgoing-/172.31.86.204-Small] | 2018-03-30 14:05:28.103000 | 172.31.88.19 | 6684 | 172.31.88.19
Adding to emp memtable [MutationStage-1] | 2018-03-30 14:05:28.103000 | 172.31.86.204 | 489 | 172.31.88.19
Enqueuing response to /172.31.88.19 [MutationStage-1] | 2018-03-30 14:05:28.114000 | 172.31.86.204 | 11273 | 172.31.88.19
Sending REQUEST_RESPONSE message to /172.31.88.19 [MessagingService-Outgoing-/172.31.88.19-Small] | 2018-03-30 14:05:28.114000 | 172.31.86.204 | 11393 | 172.31.88.19
REQUEST_RESPONSE message received from /172.31.86.204 [MessagingService-Incoming-/172.31.86.204] | 2018-03-30 14:05:28.115000 | 172.31.88.19 | 19092 | 172.31.88.19
Processing response from /172.31.86.204 [RequestResponseStage-3] | 2018-03-30 14:05:28.116000 | 172.31.88.19 | 19323 | 172.31.88.19
Request complete | 2018-03-30 14:05:28.115757 | 172.31.88.19 | 19757 | 172.31.88.19
cqlsh:admatic>
[3] 0:python*
how to trace?
sqlsh <server-ip>tracing on;INSERT INTO emp (emp_id, emp_city, emp_name, emp_phone, emp_sal) values( 4,'Pune','rajeev',9848022331, 30000);
17.tuning cassandra performance
1.using bloom filters
- Bloom filters are a performance aid for Cassandra.
- They help the database during an index scan by letting it know if an SSTable has data for a specific partition.
When clients request data, the Bloom filter checks if the row exists before the database performs disk I/O.
can set the bloom_filter_fp_chance attribute for a table to a value between 0 and 1.
- As you go from 0 to 1, you use less memory.
- A value of 0 means you set the largest value for the Bloom filter and use the highest amount of memory.
- Setting it to 1 means that you’ve disabled Bloom filters
18.compaction strategies
For the LeveledCompactionStrategy: value for the bloom_filter_fp_chance attribute 0.1
For the SizeTieredCompactionStrategy, DataTieredCompactionStrategy, and TimeWindowCompactionStrategy: value for the bloom_filter_fp_chance attribute 0.01.
- Whenever you change the value of the bloom_filter_fp_chance attribute for a table, you must regenerate the Bloom filters.
- You can regenerate the Bloom filters in two ways: you can run a manual compaction of the table or upgrade the SSTables.
manual compaction
nodetool [options] compact [(-et <end_token> | --end-token <end_token>)]
[(-s | --split-output)] [(-st <start_token> | --start-token <start_token>)] [--] [<keyspace> [<tables>...]]
[--user-defined] <SSTable file>
upgrade SStables
nodetool <options> upgradesstables
( -a | --include-all-sstables )
-- <keyspace> <table>
19.TYPES OF DATA CACHES
- Key cache
- Row cache
20.location of cached data
/var/lib/cassandra/saved_caches
21.caching a table
CREATE TABLE test (
userid text PRIMARY KEY,
first_name text,
last_name text,
)
WITH caching = { 'keys' : 'NONE', 'rows_per_partition' : '120' };
22.nodetool ring
To determine what nodes are in your ring and what state they’re in, use the ring command on nodetool.
nodetool ring
23.Installing the OpsCenter deb package
1.Modify the aptitude repository source list file
echo "deb http://debian.datastax.com/community stable main" | sudo tee -a /etc/apt/sources.list.d/datastax.community.list
2.Add the DataStax repository key to your aptitude trusted keys
curl -L http://debian.datastax.com/debian/repo_key | sudo apt-key add -
3.Install the OpsCenter package using the APT Package Manager
sudo apt-get update
sudo apt-get install opscenter -y
4.Start OpsCenter
sudo service opscenterd start
5.Go to web-browser and do the following
http://<host-ip-address>:8888/
24.nodetool upgradesstables
Rewrites SSTables on a node that are incompatible with the current version. Use this command when upgrading your server or changing compression options
nodetool upgradesstables -a
hadoop@ip-172-31-88-141:~$ nodetool upgradesstables -a
WARN 07:56:38,735 Small commitlog volume detected at /var/lib/cassandra/commitlog; setting commitlog_total_space_in_mb to 5004. You can override this in cassandra.yaml
WARN 07:56:38,739 Small cdc volume detected at /var/lib/cassandra/cdc_raw; setting cdc_total_space_in_mb to 2502. You can override this in cassandra.yaml
WARN 07:56:38,740 Only 16.235GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
examples
upgradesstables
Reads only SSTables created by old major versions of Cassandra and re-writes them to the current version one at a time
upgradesstables -a
Reads all existing SSTables and re-writes them to the current Cassandra version one at a time
upgradesstables -a keyspace table
Reads specific SSTables and re-writes them to the current Cassandra version one at a time
25.sstabledump
sstabledump /var/lib/cassandra/data/keyspace1/standard1-035107c0342011e8a6b805b724b567ac/mc-19-big-Data.db
A JSON output is given as the output
Output
"position" : 114913775, [0/1538]
"cells" : [
{ "name" : "\"C0\"", "value" : "0x430574ad3707585b42270c4fa042074ba3bef20b310c21fa427e8eaca607065e2144", "tstamp" : "2018-03-30T13:41:36.273043Z" },
{ "name" : "\"C1\"", "value" : "0xb96b5f7adc57a8c60ec145c8e71326841adcfd13c9dc333d03d4ecfe826e78c30d4e", "tstamp" : "2018-03-30T13:41:36.273043Z" },
{ "name" : "\"C2\"", "value" : "0x90dab346f4444ac9388cc9a0510d121da3724e1a205e2ed8fe9cbc69662c168758d9", "tstamp" : "2018-03-30T13:41:36.273043Z" },
{ "name" : "\"C3\"", "value" : "0xb0837c9c894a4a5359d0558874da3064eb2c942d02c60277910960b5b2ad253b25ca", "tstamp" : "2018-03-30T13:41:36.273043Z" },
{ "name" : "\"C4\"", "value" : "0x547e97419cb4d27530ce5b096b53940f17d11949e8851565630d5182623f0922b41d", "tstamp" : "2018-03-30T13:41:36.273043Z" }
]
}
]
},
{
"partition" : {
"key" : [ "37363132354b344d3031" ],
"position" : 114913775
},
"rows" : [
{
"type" : "static_block",
"position" : 114914005,
"cells" : [
{ "name" : "\"C0\"", "value" : "0xce069d8a88689d7c6259e061d56f2451f1605e1eef3e2a2e9df2c9ac1ccb6953a04c", "tstamp" : "2018-03-30T13:41:54.245006Z" },
{ "name" : "\"C1\"", "value" : "0x80c8506df4bcde677d4e5c595aa80eaef4130c9cb29d0388b5f24e6ab218a5f13ee9", "tstamp" : "2018-03-30T13:41:54.245006Z" },
{ "name" : "\"C2\"", "value" : "0xccdb2aef388e85c07d0349cc1d5203a0d74004105234376c143157aa9c336376271e", "tstamp" : "2018-03-30T13:41:54.245006Z" },
{ "name" : "\"C3\"", "value" : "0xd09ebe31c7684508bb8d91184a243c24d6810250771a92157ffb24027a7c383efadf", "tstamp" : "2018-03-30T13:41:54.245006Z" },
{ "name" : "\"C4\"", "value" : "0xdcc9cd8bebb57470d3a3e0bcd890f8b988295d512b6e8c99db777d24f30b95b87eb1", "tstamp" : "2018-03-30T13:41:54.245006Z" }
]
}
]
},
{
"partition" : {
"key" : [ "4e4f4e504e374e4f3930" ],
"position" : 114914005
},
"rows" : [
{
"type" : "static_block",
"position" : 114914235,
"cells" : [
{ "name" : "\"C0\"", "value" : "0x1d7928eb11ca117f0bd3ef3c0ad4cdc82464efc731570b8a459ef5f01f8819d63de4", "tstamp" : "2018-03-30T13:42:05.622007Z" },
{ "name" : "\"C1\"", "value" : "0xfaf9bc2b2366fdc5c43396b49b7e2c6ef9e494d80e657f3bbc42fa1d1d4ce7b34fef", "tstamp" : "2018-03-30T13:42:05.622007Z" },
{ "name" : "\"C2\"", "value" : "0x1e50b0a38d7ed82bc267cc5f7d77c95e2280a646059a8591fa2d8f4f7113a302b3d5", "tstamp" : "2018-03-30T13:42:05.622007Z" },
{ "name" : "\"C3\"", "value" : "0xfcc5624a247d9e9716d76b03275f22183e9b85d10a811fe4983d7904447b8bcaf44d", "tstamp" : "2018-03-30T13:42:05.622007Z" },
{ "name" : "\"C4\"", "value" : "0x1cc8bb866a7d27996f4c627cf295ccea03b933cdcff540e7c61d923624f4622ae9fc", "tstamp" : "2018-03-30T13:42:05.622007Z" }
]
}
]
}
]hadoop@ip-172-31-88-141:/var/lib/cassandra/data/keyspace1/standard1-035107c0342011e8a6b85b724b567ac$ sstabledump /var/lib/cassandra/data/keyspace1/standard1-035107c0342011e8a6b805b724b567ac/mc-19-big-Data.db
26.Tombstones
All the 'null' values are directly stored into something called as 'Tombstone'
Tombstones can be visualised in sstablesdump JSON output
27. nodetool drain
used before upgrading a node to a new version of Cassandra
Flushes all memtables from the node to SSTables on disk
Cassandra stops listening for connections from the client and other nodes
requires Cassandra restart after this command