Apache Ignite

GridGain Developer Hub - Apache Ignitetm

Welcome to the Apache Ignite developer hub run by GridGain. Here you'll find comprehensive guides and documentation to help you start working with Apache Ignite as quickly as possible, as well as support if you get stuck.

 

GridGain also provides Community Edition which is a distribution of Apache Ignite made available by GridGain. It is the fastest and easiest way to get started with Apache Ignite. The Community Edition is generally more stable than the Apache Ignite release available from the Apache Ignite website and may contain extra bug fixes and features that have not made it yet into the release on the Apache website.

 

Let's jump right in!

 

Documentation     Ask a Question     Download

 

Javadoc     Scaladoc     Examples
Ask A Question

Questions

1
ANSWERED

Ignite indexing performance

Hi, I have a cache with string as a key and TileKey (class below) as a value, I've noticed that when I execute a query (below) the performance is affected almost linearly by the cache size even though all the fields that are used in the query are indexed. Here is a representative benchmark - (I've used the same query (below) with the same parameters for all benchmarks) : The query returns (the same) 30 entries in all benchmarks Query on 5350 entries cache took 6-7ms Query on 10700 entries cache took 8-10ms Query on 48150 entries cache took 30-42ms Query on 96300 entries cache took 50-70ms I've executed the benchmark with 8gb single node and 4gb 2 nodes, the results were pretty much the same (in terms of query speed relatively to cache size) I'm using group index and i'm using the time field as the first group field as the time filter will reduce the result set to only 1000 entries in all benchmarks Am I doing something wrong or those are the expected performance/behavior of Ignite? Thanks Code : String strQuery = "time = ? and zoom = ? and x >= ? and x <= ? and y >= ? and y <= ?"; SqlQuery<String, TileKey> query= new SqlQuery<String, TileKey>(TileKey.class, strQuery); query.setArgs(time, zoom, xMin,xMax,yMin, yMax); QueryCursor<Entry<String, TileKey>> tileKeyCursor= tileKeyCache.query(query); Map<String, TileKey> tileKeyMap= new HashMap<String, TileKey>(); for (Entry<String, TileKey> p : keysCursor) { tileKeyMap.put(p.getKey(), p.getValue()); } Cache config : <bean class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="KeysCache" /> <property name="cacheMode" value="PARTITIONED" /> <property name="atomicityMode" value="ATOMIC" /> <property name="backups" value="0" /> <property name="queryIndexEnabled" value="true"/> <property name="indexedTypes"> <list> <value>java.lang.String</value> <value>org.ess.map.TileKey</value> </list> </property> </bean> Class : @QueryGroupIndex.List(@QueryGroupIndex(name = "idx1")) public class TileKey implements Serializable { /** * */ private static final long serialVersionUID = 1L; private String id; @QuerySqlField(index = true) @QuerySqlField.Group(name = "idx1", order = 0) private int time; @QuerySqlField(index = true) @QuerySqlField.Group(name = "idx1", order = 1) private int zoom; @QuerySqlField(index = true) @QuerySqlField.Group(name = "idx1", order = 2) private int x; @QuerySqlField(index = true) @QuerySqlField.Group(name = "idx1", order = 3) private int y; @QuerySqlField(index = true) private boolean inCache; }

Posted by Nir Avon about a year ago

1
ANSWERED

Optimistic transaction very slow on v1.3.2

Trying the below sample where I am putting 20000 entries in cache within transaction on a single node(JRE 7, heap 4GB),it takes more than 20sec to commit.Thread dumps below shows thread executing GridCacheMvccManager.addNext most of the time.The same sample with pessimistic mode takes less than 5sec Let me know if there is any other setting that should be specified or is it the best we can get considering everything is happening in memory on single node? URL url = TestCache.class.getResource("/unittest/gridgain/example-ignite.xml"); Ignite ignite = Ignition.start(url); System.out.println(); System.out.println(">>> Cache API example started."); CacheConfiguration cfg = new CacheConfiguration(); cfg.setName("myCache"); cfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL); cfg.setCacheMode(CacheMode.PARTITIONED); cfg.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED); cfg.setRebalanceMode(CacheRebalanceMode.ASYNC); cfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_ASYNC); cfg.setOffHeapMaxMemory(2 * 1024L * 1024L * 1024L); IgniteCache<String,byte[]> cache = ignite.createCache(cfg); byte[] valArr = new byte[16000]; for(int i=0;i<valArr.length;i++){ valArr[i] = 0x0; } long t1 = System.currentTimeMillis(); try (Transaction tx = (Transaction)Ignition.ignite().transactions().txStart(TransactionConcurrency.OPTIMISTIC,TransactionIsolation.REPEATABLE_READ)){ for(int i=0;i<20000;i++){ cache.put("K"+(long)i,valArr); } tx.commit(); } long t2 = System.currentTimeMillis(); System.out.println("Time taken:" + (t2-t1)); "main" prio=6 tid=0x0000000002c33800 nid=0x454 runnable [0x0000000002b4e000] java.lang.Thread.State: RUNNABLE at java.util.LinkedList.add(LinkedList.java:336) at org.apache.ignite.internal.processors.cache.GridCacheMvccManager.addNext(GridCacheMvccManager.java:733) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.addDhtLocal(GridDhtCacheEntry.java:230) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.tmLock(GridDhtCacheEntry.java:244) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.lockMultiple(IgniteTxManager.java:1586) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.prepareTx(IgniteTxManager.java:882) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userPrepare(IgniteTxLocalAdapter.java:410) at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareAsyncLocal(GridNearTxLocal.java:852)

Posted by Sajal Gupta about a year ago

1
ANSWERED

org.apache.ignite.IgniteCheckedException: Failed to serialize object: DiscoveryData

After running a single server node, a single client node was running a mesage event listener. As soon as the other second server node execution, the error occurs, What is the way to solve ? [18:32:45,592][WARN ][grid-nio-worker-1-#36%null%][TcpDiscoverySpi] Failed to ping node (status check will be initiated): bb1a4a4c-8bd0-4a36-ba22-2ba587dd4f59 [18:32:59,786][ERROR][tcp-disco-msg-worker-#5%null][TcpDiscoverySpi] Failed to marshal discovery data [comp=0, data=DiscoveryData [nodeId=d4c7c460-403a-40bd-a72a-fa10fa024049, items=[DiscoveryDataItem [routineId=0eed7a0e-57c6-4490-8e24-ea2ed431d551, prjPred=o.a.i.i.util.lang.GridFunc$72@6bbcee8a, hnd=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler@25299a21, bufSize=1, interval=0, autoUnsubscribe=false], DiscoveryDataItem [routineId=3c27f7b4-34f7-48d9-abea-3759b8e0872a, prjPred=o.a.i.i.cluster.ClusterGroupAdapter$CachesFilter@746b64d7, hnd=o.a.i.i.GridEventConsumeHandler@59c0f3e4, bufSize=1, interval=0, autoUnsubscribe=false], DiscoveryDataItem [routineId=15efa86c-0dda-4829-ba97-b74be5eb27e3, prjPred=o.a.i.i.util.lang.GridFunc$72@68777ecf, hnd=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler@7d267d9d, bufSize=1, interval=0, autoUnsubscribe=false], DiscoveryDataItem [routineId=2a4e646a-02e7-4c52-b252-c16a63de2d70, prjPred=o.a.i.i.util.lang.GridFunc$72@6a010e9b, hnd=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler@6b66187b, bufSize=1, interval=0, autoUnsubscribe=false]]]] class org.apache.ignite.IgniteCheckedException: Failed to serialize object: DiscoveryData [nodeId=d4c7c460-403a-40bd-a72a-fa10fa024049, items=[DiscoveryDataItem [routineId=0eed7a0e-57c6-4490-8e24-ea2ed431d551, prjPred=org.apache.ignite.internal.util.lang.GridFunc$72@6bbcee8a, hnd=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler@25299a21, bufSize=1, interval=0, autoUnsubscribe=false], DiscoveryDataItem [routineId=3c27f7b4-34f7-48d9-abea-3759b8e0872a, prjPred=org.apache.ignite.internal.cluster.ClusterGroupAdapter$CachesFilter@746b64d7, hnd=org.apache.ignite.internal.GridEventConsumeHandler@59c0f3e4, bufSize=1, interval=0, autoUnsubscribe=false], DiscoveryDataItem [routineId=15efa86c-0dda-4829-ba97-b74be5eb27e3, prjPred=org.apache.ignite.internal.util.lang.GridFunc$72@68777ecf, hnd=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler@7d267d9d, bufSize=1, interval=0, autoUnsubscribe=false], DiscoveryDataItem [routineId=2a4e646a-02e7-4c52-b252-c16a63de2d70, prjPred=org.apache.ignite.internal.util.lang.GridFunc$72@6a010e9b, hnd=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler@6b66187b, bufSize=1, interval=0, autoUnsubscribe=false]]] at org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal(JdkMarshaller.java:78) at org.apache.ignite.marshaller.AbstractMarshaller.marshal(AbstractMarshaller.java:51) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.collectExchangeData(TcpDiscoverySpi.java:2234) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.access$3700(TcpDiscoverySpi.java:151) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi$RingMessageWorker.processNodeAddedMessage(TcpDiscoverySpi.java:3623) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi$RingMessageWorker.processMessage(TcpDiscoverySpi.java:2630) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpiAdapter$MessageWorkerAdapter.body(TcpDiscoverySpiAdapter.java:987) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) Caused by: java.io.NotSerializableException: BISTel.PeakPerformance.Grid.GridClientController at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at org.apache.ignite.internal.GridEventConsumeHandler.writeExternal(GridEventConsumeHandler.java:314) at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$DiscoveryDataItem.writeExternal(GridContinuousProcessor.java:1673) at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at org.apache.ignite.internal.util.IgniteUtils.writeCollection(IgniteUtils.java:1366) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$DiscoveryData.writeExternal(GridContinuousProcessor.java:1601) at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal(JdkMarshaller.java:73) ... 7 more [18:32:59,787][ERROR][tcp-disco-msg-worker-#5%null][TcpDiscoverySpi] Runtime error caught during grid runnable execution: IgniteSpiThread [name=tcp-disco-msg-worker-#5%null] class org.apache.ignite.spi.IgniteSpiException: Failed to marshal discovery data. at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.collectExchangeData(TcpDiscoverySpi.java:2242) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.access$3700(TcpDiscoverySpi.java:151) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi$RingMessageWorker.processNodeAddedMessage(TcpDiscoverySpi.java:3623) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi$RingMessageWorker.processMessage(TcpDiscoverySpi.java:2630) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpiAdapter$MessageWorkerAdapter.body(TcpDiscoverySpiAdapter.java:987) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) Caused by: class org.apache.ignite.IgniteCheckedException: Failed to serialize object: DiscoveryData [nodeId=d4c7c460-403a-40bd-a72a-fa10fa024049, items=[DiscoveryDataItem [routineId=0eed7a0e-57c6-4490-8e24-ea2ed431d551, prjPred=org.apache.ignite.internal.util.lang.GridFunc$72@6bbcee8a, hnd=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler@25299a21, bufSize=1, interval=0, autoUnsubscribe=false], DiscoveryDataItem [routineId=3c27f7b4-34f7-48d9-abea-3759b8e0872a, prjPred=org.apache.ignite.internal.cluster.ClusterGroupAdapter$CachesFilter@746b64d7, hnd=org.apache.ignite.internal.GridEventConsumeHandler@59c0f3e4, bufSize=1, interval=0, autoUnsubscribe=false], DiscoveryDataItem [routineId=15efa86c-0dda-4829-ba97-b74be5eb27e3, prjPred=org.apache.ignite.internal.util.lang.GridFunc$72@68777ecf, hnd=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler@7d267d9d, bufSize=1, interval=0, autoUnsubscribe=false], DiscoveryDataItem [routineId=2a4e646a-02e7-4c52-b252-c16a63de2d70, prjPred=org.apache.ignite.internal.util.lang.GridFunc$72@6a010e9b, hnd=org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler@6b66187b, bufSize=1, interval=0, autoUnsubscribe=false]]] at org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal(JdkMarshaller.java:78) at org.apache.ignite.marshaller.AbstractMarshaller.marshal(AbstractMarshaller.java:51) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.collectExchangeData(TcpDiscoverySpi.java:2234) ... 5 more Caused by: java.io.NotSerializableException: BISTel.PeakPerformance.Grid.GridClientController at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at org.apache.ignite.internal.GridEventConsumeHandler.writeExternal(GridEventConsumeHandler.java:314) at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$DiscoveryDataItem.writeExternal(GridContinuousProcessor.java:1673) at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at org.apache.ignite.internal.util.IgniteUtils.writeCollection(IgniteUtils.java:1366) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$DiscoveryData.writeExternal(GridContinuousProcessor.java:1601) at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal(JdkMarshaller.java:73) ... 7 more

Posted by wychoi about a year ago