Spatial Data Visualization

Have been working on spatial data visualization lately.

Two interesting open source spatial tools were analysed by me for my new project. OpenGeo and GeoMondrian. I also analysed OBIEE map viewer which is a proprietary product from Oracle. Out of them OpenGeo turned out to be a pleasant surprise.

I would like my software to be like OpenGeo because of its excellent documentation and nice administrative features. I am surprised that a open source project can be so neatly executed.

I am yet to use it on a live project, but it has surely impressed me.

Configuring SAF agent

Below are the steps on how to create a SAF agent with file persistent store that receives messages from a publisher and forwards to another JMS server configured with JDBC persistent store.
Create two Domain1 and Domain2. Domain1 will be configured with SAF agent where as Domain2 will be configured with destinations where the consumer will listen to receive and process messages.

Domain2 configuration

• Create JDBC data store
• Create Persistent store based on the above JDBC data source
• Create a new JMS server based on this JDBC persistent store
• Create a new JMS module
• Create a new Subdeployment for this JMS module and choose the JMS server created above
• Create a new Connection factory
• Create a new Queue and configure this to use the subdeployment

Domain1 configuration

1. Create File based persistent store
2. Create a new JMS server based on this JDBC persistent store
3. Create a new Store-And-Forward agent.
   1. Since we are using JMS, this can only be a "Send-only" agent. 
   2. Also make sure "Logging enabled" option is selected. This allows the message to be logged in the server log file
4. Create a new JMS module
5. Create "Remote SAF Context" resource
   1. select the name of the remote SAF context
   2. provide the URL context for the SAF agent to login. This will be pointing to domain2 we created above.
   3. provide the credentials to logon to the aboe domain
6. Create "SAF Error Handling" resource
   1. choose the name
   2. set Message Handling Policy to "Always-forward" since we are neither interested in discarding any messages nor re-directing them to error destination
7. Create a new Connection factory
8. Create a new "SAF Imported Destinations" resource in the JMS module
   1. select Remote SAF Context created above
   2. select SAF Error Handling resource created in previous step
9. After creating the "SAF Imported Destinations" and saving the configuration, create required SAF destinations for the remote destinations
   1. create a new Queue
   2. choose Queue name
   3. choose Remote JNDI name. This will be the destination JNDI name configured in Domain2
   4. Save the Queue
   5. Select the Queue and provide "Local JNDI Name". This will be used by the publisher to publish any messages to this destination

This should configure the SAF agent to forward the messages successfully to the remote destinations. Now we can publish a message to the SAF destination and consume messages on the remote destination.

.

Simon Sinek: How great leaders inspire action

Start with why!!

Get Records having the 2nd highest value

SQL to fetch all records having the 2^nd highest value in a given column. This is for Oracle(pl-sql) dialect

SELECT *

FROM some_table

WHERE val=

(SELECT MAX(val)

FROM some_table

WHERE val <>

(SELECT MAX(val) FROM some_table

)

);

If you need only the 2nd largest value, then you can use

SELECT MAX(val)

FROM some_table

WHERE val <>

(SELECT MAX(val) FROM some_table

)

Similar statement holds good for 2nd smallest as well. Use MIN instead of MAX.
.

Latest transaction for all users

Here is SQL to fetch the last transaction records for all users. Note that this is for Oracle (pl/sql)

SELECT *

FROM some_table t

INNER JOIN

(SELECT user_id,

MAX(record_id ) AS record_id

FROM some_table

GROUP BY user_id

) v

ON p. record_id = v. record_id;

JMS File Store over JDBC Store

Weblogic JMS supports JDBC Store and Fle Store for persistence. Middleware uses JMS Distributed Queue backed by a JDBC store. This has led to some issues related to the corruption & crash of the Persistent store.

Problem faced:

There were a few instances of JDBC Persistent store crashes, due to the inability to connect to the JDBC store DB server. Either the persistent store was corrupted (meaning the store's DB tables had to be purged & the JMS restarted to bring the JMS back in action) or the JDBC Persistent store just crashed (required a JMS server restart for normal functioning) . Many messages were lost as a result of these issues and the reliability was compromised.

Issues & Impacts

• All messages in the store were lost since the table had to be purged inorder to recover.
• All incoming messages during the failure period were lost since the JMS was not in a state to receive incoming messages (in TRANSACTION_SEND state).
• The messages consumed by the listener (during the crash period) could not be committed or rolled back (in TRANSACTION_RECD state) were also lost after a restart. In this case, if the message was supposed to rollback, it was not.
• A restart was always required for the JMS server to funtion.

All this behavior were also simulated in the lab and confirmed.

In a nutshell, eventhough the JDBC store is more scalable - the reliability is compromised due to the Persistent store crashes. The database connectivity was not very reliable / high (due to DB unavailability & network connectivity). We used Oracle RAC (for high availability, and still the persistent store crashed - did not failover to the other RAC node). This could be a RAC related issue too. So, based on these behavioral aspects - JDBC store seems less reliable.

Alternate approaches tried in the lab:

• Using a SAF agent with a JDBC persistent store: We were able to get this configuration working. This forces the SAF agent & the actual JMS to be on different servers. Also, SAF did not seem to be a right fit for this use case since the caller OSB and JMS are on the same Weblogic Domain/Server. This approach may still fail when a listener tries to commit or rollback during the JMS Persistent store crash period.
• Use a JMS File Store for the Distributed Queue: The File Store seems to be more Reliable based on the trends seen so far. Since the File store seems more reliable, the Issues & Impacts mentioned above seems alleviated / negligible. Scalability could be an issue compared to the JDBC store - however the current file system configuration should support the expected volume in the near future.
  

Weblogic documentation on File Store vs JDBC Store:

http://download.oracle.com/docs/cd/E12840_01/wls/docs103/config_wls/store.html

Comparing File Stores and JDBC Stores

The following are some similarities and differences between file stores and JDBC stores:

• The default persistent store can only be a file store. Therefore, a JDBC store cannot be used as a default persistent store.
• The transaction log (TLOG) can only be stored in a default store.
• Both have the same transaction semantics and guarantees. As with JDBC store writes, file store writes are guaranteed to be persisted to disk and are not simply left in an intermediate (that is, unsafe) cache.
• Both have the same application interface (no difference in application code).
• All things being equal, file stores generally offer better throughput than a JDBC store. | Note: | If a database is running on high-end hardware with very fast disks, and WebLogic Server is running on slower hardware or with slower disks, then you may get better performance from the JDBC store. |
• File stores are generally easier to configure and administer, and do not require that WebLogic subsystems depend on any external component.
• File stores generate no network traffic; whereas, JDBC stores will generate network traffic if the database is on a different machine from WebLogic Server.
• JDBC stores may make it easier to handle failure recovery since the JDBC interface can access the database from any machine on the same network. With the file store, the disk must be shared or migrated.
  

Conclusion:
The solution can vary based on the infrastructure and needs of the project. For our project File store was suggested, so that the reliability can be higher.

From a monitoring perspective, there should be a disk space alert if the file system disk space reaches 75% limit.

SoapUI and Maven integration

Configure modules with the following configuration for SoapUI and Maven integration.

1. SOAPUI change: Make a Test Suite based on a feature. Please note that users must have a TestSuite with all their SOAP Requests.

2. Note the location of the ***soapui-project.xml file location. We need to provide this path inside the pom.xml for the main project.

3. Navigate to /gdev/home/.m2 and open settings.xml file. Check if you have the url for nexus thirdparty repository. If not, add the below configuration inside :

<repository>

<id>local1</id>

<url>http://hostname:8081/nexus/content/repositories/thirdparty</url>

<releases><enabled>true</enabled></releases>

<snapshots><enabled>false</enabled></snapshots>

</repository>

4. Change the main project pom.xml of your project to include the following in plugins. This is an example, so please do your change according to your environment.

<plugin>

<groupId>eviware</groupId>

<artifactId>maven-soapui-plugin</artifactId>

<version>2.5.1</version>

<configuration>

<projectFile>/dev/software/soapUI_Projects/4-22-2011_proj/proj-4-22-2011-soapui-project.xml</projectFile>

<host>http://mwdev1.company.com:7001</host>

<endpoint>http://mwdev1.company.com:7001/project-1.0.0/services</endpoint>

<outputFolder>/dev/software/soapUI_Projects/4-22-2011_proj/</outputFolder>

<junitReport>true</junitReport>

<printReport>true</printReport>

<exportAll>true</exportAll>

</configuration>

</plugin>

5. Run maven command , "mvn eviware:maven-soapui-plugin:test".

6. Report could be found in outputFolder.

7. SVN change: Check-in the soapui-project file (e.g. proj-4-22-2011-soapui-project.xml).

8. More info can be found at : http://www.soapui.org/Test-Automation/maven-2x.html.

Note: If you get the OutOfMemory error, please do the following on the linux console.

echo $PATH

From the above command check the maven installation directory and edit mvn in bin folder.
Add export MAVEN_OPTS=-Xmx512m at the first line after comments section.