Cassandra timeout during write query at consistency ONE

 

1         Symptoms

The OGG report file show   timout exception of connecting to Cassandra, so the OGG replicat was abended.

 

2019-03-25 11:20:36  INFO    OGG-06510  Using the following key columns for target table APMT_CONTAINER_SERVICE_BLUE.CONTAINER_HISTORY: TERMINAL_ID, CONTAINER_NUMBER, EVENT_GKEY, VISIT_STATE.

 

              368147 records processed as of 2019-03-25 11:20:37 (rate 0,delta 0)

              639799 records processed as of 2019-03-25 11:21:37 (rate 1,delta 4527)

=ERROR 2019-03-25 11:22:12.000088 [main] – Async statement execution on Cassandra failed for table [APMT_CONTAINER_SERVICE_BLUE.CONTAINER_HISTORY] at position [00000000040140354920].

com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)

        at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:73) ~[cassandra-driver-core-3.1.3.jar:?]

        at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:26) ~[cassandra-driver-core-3.1.3.jar:?]

        at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) ~[cassandra-driver-core-3.1.3.jar:?]

        at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245) ~[cassandra-driver-core-3.1.3.jar:?]

        at oracle.goldengate.handler.cassandra.session.CassandraAsyncSession.flush(CassandraAsyncSession.java:53) [ggcass-12.3.2.1.0.018.jar:12.3.2.1.0.018]

        at oracle.goldengate.handler.cassandra.CassandraTablesManager.transactionCommit(CassandraTablesManager.java:98) [ggcass-12.3.2.1.0.018.jar:12.3.2.1.0.018]

        at oracle.goldengate.handler.cassandra.CassandraHandler.transactionCommit(CassandraHandler.java:179) [ggcass-12.3.2.1.0.018.jar:12.3.2.1.0.018]

        at oracle.goldengate.datasource.DsEventManager$2.send(DsEventManager.java:322) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]

        at oracle.goldengate.datasource.DsEventManager.distributeEvent(DsEventManager.java:110) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]

        at oracle.goldengate.datasource.DsEventManager.fireTransactionCommit(DsEventManager.java:330) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]

        at oracle.goldengate.datasource.AbstractDataSource.fireTransactionCommit(AbstractDataSource.java:511) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]

        at oracle.goldengate.datasource.UserExitDataSource.commitTransaction(UserExitDataSource.java:2000) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]

        at oracle.goldengate.datasource.UserExitDataSource.commitTx(UserExitDataSource.java:1690) [ggdbutil-12.3.2.1.0.018.jar:12.3.2.1.0.018]

Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)

        at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:100) ~[cassandra-driver-core-3.1.3.jar:?]

        at com.datastax.driver.core.Responses$Error.asException(Responses.java:122) ~[cassandra-driver-core-3.1.3.jar:?]

        at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:504) ~[cassandra-driver-core-3.1.3.jar:?]

        at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1070) ~[cassandra-driver-core-3.1.3.jar:?]

        at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:993) ~[cassandra-driver-core-3.1.3.jar:?]

        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266) ~[netty-handler-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) ~[netty-common-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145) ~[netty-common-4.0.37.Final.jar:4.0.37.Final]

        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]

Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)

        at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:59) ~[cassandra-driver-core-3.1.3.jar:?]

        at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:37) ~[cassandra-driver-core-3.1.3.jar:?]

        at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:289) ~[cassandra-driver-core-3.1.3.jar:?]

        at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:269) ~[cassandra-driver-core-3.1.3.jar:?]

        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) ~[netty-codec-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391) ~[netty-transport-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) ~[netty-common-4.0.37.Final.jar:4.0.37.Final]

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145) ~[netty-common-4.0.37.Final.jar:4.0.37.Final]

        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]

Exception in thread “main” oracle.goldengate.util.GGException: Error detected handling transaction commit event.

        at oracle.goldengate.datasource.UserExitDataSource.commitTransaction(UserExitDataSource.java:2002)

        at oracle.goldengate.datasource.UserExitDataSource.commitTx(UserExitDataSource.java:1690)

 

Source Context :

  SourceModule            : [gglib.ggdal.adapter.java]

  SourceID                : [/scratch/aime/adestore/views/aime_adc4150505/oggcore/OpenSys/src/gglib/ggdal/Adapter/Java/JavaAdapter.cpp]

  SourceMethod            : [HandleJavaException]

  SourceLine              : [246]

  ThreadBacktrace         : [20] elements

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f67aa83e0ae]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, …)+0x6ac) [0x7f67aa82e9bc]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/libgglog.so(_MSG_String(CSourceContext*, int, char const*, CMessageFactory::MessageDisposition)+0x39) [0x7f67aa81cd19]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/libggjava.so(+0x2e9a7) [0x7f67a04e19a7]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/libggjava.so(ggs::gglib::ggdal::CJavaRecordWriter::CommitTransaction(ggs::gglib::ggdal::CDALError&)+0x13d) [0x7f67a04e53ed]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::gglib::ggdal::CDALRecordWriter::CommitTransaction(ggs::gglib::ggdal::CDALError&)+0x4d) [0x82e92d]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(GenericImpl::CommitTransaction()+0x28) [0x8162e8]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(odbc_commit_transaction()+0x16) [0x80dbf6]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(Commit_and_Checkpoint(short, bool&, RepCsn&)+0x3d5) [0x6e3115]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ClassicWorker::CommitAndCheckpoint(short)+0x2e) [0x5b6dde]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(REP_end_transaction(short)+0x17) [0x6e2a57]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::er::ReplicatContext::processReplicatLoop()+0x4897) [0x631157]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::er::ReplicatContext::run()+0x45) [0x60ff85]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat() [0x6e4aaf]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x5e) [0x7e27ce]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x173) [0x7e6b93]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(ggs::gglib::MultiThreading::MainThread::Run(int, char**)+0x140) [0x7e7400]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat(main+0x3b) [0x6e784b]]

                          : [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f67a2ecd3d5]]

                          : [/ogg/app/oracle/ogg4bd/12.3.2.1.0/replicat() [0x550271]]

 

2019-03-25 11:22:12  ERROR   OGG-15051  Java or JNI exception:

oracle.goldengate.util.GGException: Error detected handling transaction commit event.

 

***********************************************************************

*                   ** Run Time Statistics **                         *

***********************************************************************

 

Last record for the last committed transaction is the following:

___________________________________________________________________

Trail name :  /ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/TR000000004

Hdr-Ind    :     E  (x45)     Partition  :     .  (x0c)

UndoFlag   :     .  (x00)     BeforeAfter:     A  (x41)

RecLength  :   193 (x00c1)    IO Time    : 2019-03-20 16:04:24.000000

IOType     :     5  (x05)     OrigNode   :   255  (xff)

TransInd   :     .  (x02)     FormatType :     R  (x52)

SyskeyLen  :     0  (x00)     Incomplete :     .  (x00)

AuditRBA   :       1354       AuditPos   : 867749536

Continued  :     N  (x00)     RecCount   :     1  (x01)

 

2019-03-20 16:04:24.000000 Insert             Len   193 RBA 50584192

TDR Index: 1

___________________________________________________________________

 

Reading /ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/TR000000004, current RBA 140423077, 884646 records, m_file_seqno = 4, m_file_rba = 140423077

Report at 2019-03-25 11:22:12 (activity since 2019-03-20 16:04:31)

Last log location read:

     FILE:      /ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/TR000000004

     SEQNO:     4

     RBA:       140423077

     TIMESTAMP: 2019-03-25 11:20:15.000000

     EOF:       NO

     READERR:   0

2019-03-25 11:22:12  ERROR   OGG-01668  PROCESS ABENDING.

 

 

2         Debug

2.1        restart replicat

If restart replicat process, it will continue to send data to Cassandra but get abended again after for a while.

GGSCI (linuxvm01) 12>   start RCEC

REPLICAT   RCEC  Last Started 2019-03-26 08:24   Status RUNNING

Checkpoint Lag       21:04:00 (updated 00:00:00 ago)

Process ID           45716

Log Read Checkpoint  File /ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/TR000000004

                     2019-03-25 11:20:15.000000  RBA 157121558

 

2.2        Lots of trail file reach to maximum size

oracle@linuxvm01:/home/oracle %

cd /ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/

 

oracle@linuxvm01:/ogg/app/oracle/oggma/12.3.0.1.4/DEPLOYMENT_LOCAL/LOCAL/var/lib/data/% ls -lrt

total 607288

–rw-r—–. 1 oracle dba      1566 Mar 19 14:46 TR000000003

-rw-r—–. 1 oracle dba      1566 Mar 20 12:13 KA000000002

-rw-r—–. 1 oracle dba      1566 Mar 20 12:13 KA000000003

-rw-r—–. 1 oracle dba 199999954 Mar 25 11:21 TR000000004

-rw-r—–. 1 oracle dba 199999904 Mar 25 11:22 TR000000005

-rw-r—–. 1 oracle dba 199999707 Mar 25 11:22 TR000000006

-rw-r—–. 1 oracle dba   8989574 Mar 25 11:22 TR000000007

2.3         Check the props file and parameter file

 [oracle@linuxvm01 ]$ more cass.props

gg.handlerlist=cassandra

 

#The handler properties

gg.handler.cassandra.type=cassandra

gg.handler.cassandra.mode=op

gg.handler.cassandra.contactPoints=cassandra01

gg.handler.cassandra.username=cass_account

gg.handler.cassandra.password=cass_password

# API093, removed options to stop SOLR_QUERY being dropped by OGG

# gg.handler.cassandra.ddlHandling=CREATE,ADD,DROP

gg.handler.cassandra.compressedUpdates=false

gg.handler.cassandra.cassandraMode=async

gg.handler.cassandra.consistencyLevel=ONE

 

#Avro Metadata Provider Properties

gg.mdp.type=avro

gg.mdp.schemaFilesPath=/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/avromdp

gg.mdp.charset=UTF-8

 

goldengate.userexit.writers=javawriter

javawriter.stats.display=TRUE

javawriter.stats.full=TRUE

 

gg.log=log4j

gg.log.level=INFO

 

gg.report.time=30sec

 

#Set the classpath here to the Datastax Cassandra Java Driver (3.1 latest)

#Link to the Cassandra drivers website

#http://cassandra.apache.org/doc/latest/getting_started/drivers.html#java

#Link to the Datastax Cassandra Java Driver

#https://github.com/datastax/java-driver

gg.classpath=/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/drivers/cassandra/cassandra-java-driver-3.1.3/*:/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/drivers/cassandra/cassandra-java-driver-3.1.3/lib/*:/ogg/app/oracle/ogg4bd/

12.3.2.1.0/ggjava/lib/*:

 

javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm

 

[oracle@linuxvm01 ]$ more ../REPLICAT_COMMON_PARAMS_OGG4BD_EVENT_CASSANDRA.inc

———– Start of Global INCLUDE file ————

COMMENT Global Parameters Include file (for all processes)

 

DISCARDFILE ?OGG_DEPLOYMENT/dirrpt/?OGG_GROUP.dsc, APPEND, MEGABYTES 256

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 100

MAXTRANSOPS 500

ALLOWDUPTARGETMAP

 

———– End of Global INCLUDE file ————

 

3          Solution

oracle@linuxvm01:/ogg/app/oracle/ogg4bd/12.3.2.1.0/dirprm % more REPLICAT_COMMON_PARAMS_OGG4BD_EVENT_CASSANDRA.inc

———– Start of Global INCLUDE file ————

COMMENT Global Parameters Include file (for all processes)

 

DISCARDFILE ?OGG_DEPLOYMENT/dirrpt/?OGG_GROUP.dsc, APPEND, MEGABYTES 256

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 20    –old value: 100

MAXTRANSOPS 50        –old value: 500

ALLOWDUPTARGETMAP

oracle@linuxvm01:/ogg/app/oracle/ogg4bd/12.3.2.1.0/dirprm % more TRss.props

gg.handlerlist=cassandra

 

#The handler properties

gg.handler.cassandra.type=cassandra

gg.handler.cassandra.mode=op

gg.handler.cassandra.contactPoints=cassandra01

gg.handler.cassandra.username=cass_account

gg.handler.cassandra.password=cass_password

# API093, removed options to stop SOLR_QUERY being dropped by OGG

# gg.handler.cassandra.ddlHandling=CREATE,ADD,DROP

gg.handler.cassandra.compressedUpdates=false

gg.handler.cassandra.cassandraMode=async

gg.handler.cassandra.consistencyLevel=ANY   –old value: ONE

 

#Avro Metadata Provider Properties

gg.mdp.type=avro

gg.mdp.schemaFilesPath=/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/avromdp

gg.mdp.charset=UTF-8

 

goldengate.userexit.writers=javawriter

javawriter.stats.display=TRUE

javawriter.stats.full=TRUE

 

gg.log=log4j

gg.log.level=INFO

 

gg.report.time=30sec

 

#Set the classpath here to the Datastax Cassandra Java Driver (3.1 latest)

#Link to the Cassandra drivers website

#http://cassandra.apache.org/doc/latest/getting_started/drivers.html#java

#Link to the Datastax Cassandra Java Driver

#https://github.com/datastax/java-driver

gg.classpath=/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/drivers/cassandra/cassandra-java-driver-3.1.3/*:/ogg/app/oracle/ogg4bd/12.3.2.1.0/adapters/drivers/cassandra/cassandra-java-driver-3.1.3/lib/*:/ogg/app/oracle/ogg4bd/

12.3.2.1.0/ggjava/lib/*:

 

javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm

 

4         Reference

Write Timeout Exception

When running the Cassandra handler, you may experience a com.datastax.driver.core.exceptions.WriteTimeoutException exception that causes the Replicat process to abend. It is likely to occur under some or all of the following conditions:

  • The Cassandra Handler processes large numbers of operations, putting the Cassandra cluster under a significant processing load.
  • GROUPTRANSOPS is configured higher than the value of 1000 default.
  • The Cassandra Handler is configured in asynchronous mode.
  • The Cassandra Handler is configured with a consistency level higher than ONE.

When this problem occurs, the Cassandra Handler is streaming data faster than the Cassandra cluster can process it. The write latency in the Cassandra cluster finally exceeds the write request timeout period, which in turn results in the exception.

The following are potential solutions:

  • Increase the write request timeout period. This is controlled with the write_request_timeout_in_ms property in Cassandra and is located in the cassandra.yaml file in the cassandra_install/conf directory. The default is 2000 (2 seconds). You can increase this value to move past the error, and then restart the Cassandra node or nodes for the change to take effect.
  • Decrease the GROUPTRANSOPS configuration value of the Replicat process. Typically, decreasing the GROUPTRANSOPSconfiguration decreases the size of transactions processed and reduces the likelihood that the Cassandra Handler can overtax the Cassandra cluster.
  • Reduce the consistency level of the Cassandra Handler. This in turn reduces the amount of work the Cassandra cluster has to complete for an operation to be considered as written.

 https://docs.oracle.com/en/middleware/goldengate/big-data/12.3.2.1/gadbd/using-cassandra-handler.html#GUID-9E1A7E82-47E6-475B-887A-B8FE3A511389

https://www.datastax.com/dev/blog/how-cassandra-deals-with-replica-failure 

Leave a Reply

Your email address will not be published. Required fields are marked *