1 Symptoms
The OGG report file show timout exception of connecting to Cassandra, so the OGG replicat was abended.
2019-03-25 11:20:36 INFO OGG-06510 Using the following key columns for target table APMT_CONTAINER_SERVICE_BLUE.CONTAINER_HISTORY: TERMINAL_ID, CONTAINER_NUMBER, EVENT_GKEY, VISIT_STATE.
368147 records processed as of 2019-03-25 11:20:37 (rate 0,delta 0)
639799 records processed as of 2019-03-25 11:21:37 (rate 1,delta 4527)
=ERROR 2019-03-25 11:22:12.000088 [main] – Async statement execution on Cassandra failed for table [APMT_CONTAINER_SERVICE_BLUE.CONTAINER_HISTORY] at position [00000000040140354920].
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)
at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:73) ~[cassandra-driver-core-3.1.3.jar:?]
at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:26) ~[cassandra-driver-core-3.1.3.jar:?]
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) ~[cassandra-driver-core-3.1.3.jar:?]
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245) ~[cassandra-driver-core-3.1.3.jar:?]
at oracle.goldengate.handler.cassandra.session.CassandraAsyncSession.flush(CassandraAsyncSession.java:53) [ggcass-12.3.2.1.0.018.jar:12.3.2.1.0.018]
at oracle.goldengate.handler.cassandra.CassandraTablesManager.transactionCommit(CassandraTablesManager.java:98) [ggcass-12.3.2.1.0.018.jar:12.3.2.1.0.018]
at oracle.goldengate.handler.cassandra.CassandraHandler.transactionCommit(CassandraHandler.java:179) [ggcass-12.3.2.1.0.018.jar:12.3.2.1.0.018]
at oracle.goldengate.datasource.DsEventManager$2.send(DsEventManager.java:322) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]
at oracle.goldengate.datasource.DsEventManager.distributeEvent(DsEventManager.java:110) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]
at oracle.goldengate.datasource.DsEventManager.fireTransactionCommit(DsEventManager.java:330) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]
at oracle.goldengate.datasource.AbstractDataSource.fireTransactionCommit(AbstractDataSource.java:511) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]
at oracle.goldengate.datasource.UserExitDataSource.commitTransaction(UserExitDataSource.java:2000) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]
at oracle.goldengate.datasource.UserExitDataSource.commitTx(UserExitDataSource.java:1690) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]
Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)
at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:100) ~[cassandra-driver-core-3.1.3.jar:?]
at com.datastax.driver.core.Responses$Error.asException(Responses.java:122) ~[cassandra-driver-core-3.1.3.jar:?]
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:504) ~[cassandra-driver-core-3.1.3.jar:?]
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1070) ~[cassandra-driver-core-3.1.3.jar:?]
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:993) ~[cassandra-driver-core-3.1.3.jar:?]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266) ~[netty-handler-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) ~[netty-common-4.0.37.Final.jar:4.0.37.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145) ~[netty-common-4.0.37.Final.jar:4.0.37.Final]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]
Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:59) ~[cassandra-driver-core-3.1.3.jar:?]
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:37) ~[cassandra-driver-core-3.1.3.jar:?]
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:289) ~[cassandra-driver-core-3.1.3.jar:?]
at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:269) ~[cassandra-driver-core-3.1.3.jar:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) ~[netty-common-4.0.37.Final.jar:4.0.37.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145) ~[netty-common-4.0.37.Final.jar:4.0.37.Final]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]
Exception in thread “main” oracle.goldengate.util.GGException: Error detected handling transaction commit event.
at oracle.goldengate.datasource.UserExitDataSource.commitTransaction(UserExitDataSource.java:2002)
at oracle.goldengate.datasource.UserExitDataSource.commitTx(UserExitDataSource.java:1690)
Source Context :
SourceModule : [gglib.ggdal.adapter.java]
SourceID : [/scratch/aime/adestore/views/aime_adc4150505/oggcore/OpenSys/src/gglib/ggdal/Adapter/Java/JavaAdapter.cpp]
SourceMethod : [HandleJavaException]
SourceLine : [246]
ThreadBacktrace : [20] elements
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f67aa83e0ae]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, …)+0x6ac) [0x7f67aa82e9bc]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/libgglog.so(_MSG_String(CSourceContext*, int, char const*, CMessageFactory::MessageDisposition)+0x39) [0x7f67aa81cd19]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/libggjava.so(+0x2e9a7) [0x7f67a04e19a7]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/libggjava.so(ggs::gglib::ggdal::CJavaRecordWriter::CommitTransaction(ggs::gglib::ggdal::CDALError&)+0x13d) [0x7f67a04e53ed]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::gglib::ggdal::CDALRecordWriter::CommitTransaction(ggs::gglib::ggdal::CDALError&)+0x4d) [0x82e92d]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(GenericImpl::CommitTransaction()+0x28) [0x8162e8]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(odbc_commit_transaction()+0x16) [0x80dbf6]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(Commit_and_Checkpoint(short, bool&, RepCsn&)+0x3d5) [0x6e3115]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ClassicWorker::CommitAndCheckpoint(short)+0x2e) [0x5b6dde]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(REP_end_transaction(short)+0x17) [0x6e2a57]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::er::ReplicatContext::processReplicatLoop()+0x4897) [0x631157]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::er::ReplicatContext::run()+0x45) [0x60ff85]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat() [0x6e4aaf]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x5e) [0x7e27ce]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x173) [0x7e6b93]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::gglib::MultiThreading::MainThread::Run(int, char**)+0x140) [0x7e7400]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(main+0x3b) [0x6e784b]]
: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f67a2ecd3d5]]
: [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat() [0x550271]]
2019-03-25 11:22:12 ERROR OGG-15051 Java or JNI exception:
oracle.goldengate.util.GGException: Error detected handling transaction commit event.
***********************************************************************
* ** Run Time Statistics ** *
***********************************************************************
Last record for the last committed transaction is the following:
___________________________________________________________________
Trail name : /ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/TR000000004
Hdr-Ind : E (x45) Partition : . (x0c)
UndoFlag : . (x00) BeforeAfter: A (x41)
RecLength : 193 (x00c1) IO Time : 2019-03-20 16:04:24.000000
IOType : 5 (x05) OrigNode : 255 (xff)
TransInd : . (x02) FormatType : R (x52)
SyskeyLen : 0 (x00) Incomplete : . (x00)
AuditRBA : 1354 AuditPos : 867749536
Continued : N (x00) RecCount : 1 (x01)
2019-03-20 16:04:24.000000 Insert Len 193 RBA 50584192
TDR Index: 1
___________________________________________________________________
Reading /ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/TR000000004, current RBA 140423077, 884646 records, m_file_seqno = 4, m_file_rba = 140423077
Report at 2019-03-25 11:22:12 (activity since 2019-03-20 16:04:31)
Last log location read:
FILE: /ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/TR000000004
SEQNO: 4
RBA: 140423077
TIMESTAMP: 2019-03-25 11:20:15.000000
EOF: NO
READERR: 0
2019-03-25 11:22:12 ERROR OGG-01668 PROCESS ABENDING.
2 Debug
2.1 restart replicat
If restart replicat process, it will continue to send data to Cassandra but get abended again after for a while.
GGSCI (linuxvm01) 12> start RCEC
REPLICAT RCEC Last Started 2019-03-26 08:24 Status RUNNING
Checkpoint Lag 21:04:00 (updated 00:00:00 ago)
Process ID 45716
Log Read Checkpoint File /ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/TR000000004
2019-03-25 11:20:15.000000 RBA 157121558
2.2 Lots of trail file reach to maximum size
oracle@linuxvm01:/home/oracle %
cd /ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/
oracle@linuxvm01:/ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/% ls -lrt
total 607288
–rw-r—–. 1 oracle dba 1566 Mar 19 14:46 TR000000003
-rw-r—–. 1 oracle dba 1566 Mar 20 12:13 KA000000002
-rw-r—–. 1 oracle dba 1566 Mar 20 12:13 KA000000003
-rw-r—–. 1 oracle dba 199999954 Mar 25 11:21 TR000000004
-rw-r—–. 1 oracle dba 199999904 Mar 25 11:22 TR000000005
-rw-r—–. 1 oracle dba 199999707 Mar 25 11:22 TR000000006
-rw-r—–. 1 oracle dba 8989574 Mar 25 11:22 TR000000007
2.3 Check the props file and parameter file
[oracle@linuxvm01 ]$ more cass.props
gg.handlerlist=cassandra
#The handler properties
gg.handler.cassandra.type=cassandra
gg.handler.cassandra.mode=op
gg.handler.cassandra.contactPoints=cassandra01
gg.handler.cassandra.username=cass_account
gg.handler.cassandra.password=cass_password
# API093, removed options to stop SOLR_QUERY being dropped by OGG
# gg.handler.cassandra.ddlHandling=CREATE,ADD,DROP
gg.handler.cassandra.compressedUpdates=false
gg.handler.cassandra.cassandraMode=async
gg.handler.cassandra.consistencyLevel=ONE
#Avro Metadata Provider Properties
gg.mdp.type=avro
gg.mdp.schemaFilesPath=/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/avromdp
gg.mdp.charset=UTF-8
goldengate.userexit.writers=javawriter
javawriter.stats.display=TRUE
javawriter.stats.full=TRUE
gg.log=log4j
gg.log.level=INFO
gg.report.time=30sec
#Set the classpath here to the Datastax Cassandra Java Driver (3.1 latest)
#Link to the Cassandra drivers website
#http://cassandra.apache.org/doc/latest/getting_started/drivers.html#java
#Link to the Datastax Cassandra Java Driver
#https://github.com/datastax/java-driver
gg.classpath=/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/drivers/cassandra/cassandra-java-driver-3.1.3/*:/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/drivers/cassandra/cassandra-java-driver-3.1.3/lib/*:/ogg/app/oracle/ogg4bd/
12.3.2.1.0/ggjava/lib/*:
javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm
[oracle@linuxvm01 ]$ more ../REPLICAT_COMMON_PARAMS_OGG4BD_EVENT_CASSANDRA.inc
———– Start of Global INCLUDE file ————
COMMENT Global Parameters Include file (for all processes)
DISCARDFILE ?OGG_DEPLOYMENT/dirrpt/?OGG_GROUP.dsc, APPEND, MEGABYTES 256
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 100
MAXTRANSOPS 500
ALLOWDUPTARGETMAP
———– End of Global INCLUDE file ————
3 Solution
oracle@linuxvm01:/ogg/app/oracle/ogg4bd/12.3.2.1.0/dirprm % more REPLICAT_COMMON_PARAMS_OGG4BD_EVENT_CASSANDRA.inc
———– Start of Global INCLUDE file ————
COMMENT Global Parameters Include file (for all processes)
DISCARDFILE ?OGG_DEPLOYMENT/dirrpt/?OGG_GROUP.dsc, APPEND, MEGABYTES 256
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 20 –old value: 100
MAXTRANSOPS 50 –old value: 500
ALLOWDUPTARGETMAP
oracle@linuxvm01:/ogg/app/oracle/ogg4bd/12.3.2.1.0/dirprm % more TRss.props
gg.handlerlist=cassandra
#The handler properties
gg.handler.cassandra.type=cassandra
gg.handler.cassandra.mode=op
gg.handler.cassandra.contactPoints=cassandra01
gg.handler.cassandra.username=cass_account
gg.handler.cassandra.password=cass_password
# API093, removed options to stop SOLR_QUERY being dropped by OGG
# gg.handler.cassandra.ddlHandling=CREATE,ADD,DROP
gg.handler.cassandra.compressedUpdates=false
gg.handler.cassandra.cassandraMode=async
gg.handler.cassandra.consistencyLevel=ANY –old value: ONE
#Avro Metadata Provider Properties
gg.mdp.type=avro
gg.mdp.schemaFilesPath=/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/avromdp
gg.mdp.charset=UTF-8
goldengate.userexit.writers=javawriter
javawriter.stats.display=TRUE
javawriter.stats.full=TRUE
gg.log=log4j
gg.log.level=INFO
gg.report.time=30sec
#Set the classpath here to the Datastax Cassandra Java Driver (3.1 latest)
#Link to the Cassandra drivers website
#http://cassandra.apache.org/doc/latest/getting_started/drivers.html#java
#Link to the Datastax Cassandra Java Driver
#https://github.com/datastax/java-driver
gg.classpath=/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/drivers/cassandra/cassandra-java-driver-3.1.3/*:/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/drivers/cassandra/cassandra-java-driver-3.1.3/lib/*:/ogg/app/oracle/ogg4bd/
12.3.2.1.0/ggjava/lib/*:
javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm
4 Reference
Write Timeout Exception
When running the Cassandra handler, you may experience a com.datastax.driver.core.exceptions.WriteTimeoutException exception that causes the Replicat process to abend. It is likely to occur under some or all of the following conditions:
- The Cassandra Handler processes large numbers of operations, putting the Cassandra cluster under a significant processing load.
- GROUPTRANSOPS is configured higher than the value of 1000 default.
- The Cassandra Handler is configured in asynchronous mode.
- The Cassandra Handler is configured with a consistency level higher than ONE.
When this problem occurs, the Cassandra Handler is streaming data faster than the Cassandra cluster can process it. The write latency in the Cassandra cluster finally exceeds the write request timeout period, which in turn results in the exception.
The following are potential solutions:
- Increase the write request timeout period. This is controlled with the write_request_timeout_in_ms property in Cassandra and is located in the cassandra.yaml file in the cassandra_install/conf directory. The default is 2000 (2 seconds). You can increase this value to move past the error, and then restart the Cassandra node or nodes for the change to take effect.
- Decrease the GROUPTRANSOPS configuration value of the Replicat process. Typically, decreasing the GROUPTRANSOPSconfiguration decreases the size of transactions processed and reduces the likelihood that the Cassandra Handler can overtax the Cassandra cluster.
- Reduce the consistency level of the Cassandra Handler. This in turn reduces the amount of work the Cassandra cluster has to complete for an operation to be considered as written.
https://www.datastax.com/dev/blog/how-cassandra-deals-with-replica-failure
