research-article

Anomaly detection in agri warehouse construction

Authors:

Andrew McCarren,

Suzanne McCarthy,

Conor O Sullivan,

Mark RoantreeAuthors Info & Claims

ACSW '17: Proceedings of the Australasian Computer Science Week Multiconference

Article No.: 17, Pages 1 - 10

https://doi.org/10.1145/3014812.3014829

Published: 31 January 2017 Publication History

Abstract

As with many sectors, strategists and decision makers in the agricultural sector have a requirement to predict key measures such as product and feed pricing in order to maintain their position and, in some cases, to survive in their industry. Predictive algorithms in the area of Agri Analytics have shown to be very difficult due to the wide range of parameters and often unpredictable nature of some of these variables. Improving the predictive capability of Agri planners requires access to up-to-date external information in addition to the analyses provided by their own in-house databases. This motivates the need for an Agri Data Ware-house together with appropriate cleaning and transformation processes. However, with the availability of rich and wide ranging sources of Agri data now available online, there is a strong motivation to process as much current, online information as possible. In this work, we introduce the Agri Data Warehouse built for the DATAS project which not only harvests from a large number of online sources but also adopts an anomaly detection and labelling process to assist transformation and loading into the warehouse.

References

[1]

AUC. 2016. http://gim.unmc.edu/dxtests/roc3.htm.

[2]

Beautiful Soup Documentation. 2016. https://www.crummy.com/software/BeautifulSoup/bs4/doc/.

[3]

Cox D.R. 1958. The Regression Analysis of Binary Sequences. Journal of the Royal Statistical Society. Series B (Methodological), 20,2, 215--242.

[4]

Stonebraker M. and Brodie M. 1995. Migrating Legacy Systems: Gateways, Interfaces and the Incremental Approach. Morgan Kaufmann.

Digital Library

[5]

CLAL. 2016. http://www.clal.it/en/.

[6]

Cavallo A. and Rigobon R. 2016. The Billion Prices Project: Using Online Prices for Measurement and Research. Journal of Economic Perspectives. 30, 2, 151--178.

[7]

Shearer C. 2000. The CRISP-DM Model:The New Blueprint for Data Mining. Journal of Data Warehousing. 5,4,13--22.

[8]

Chandola V., Banerjee A. and Kumar V. 2009. Anomaly Detection: A Survey. ACM Computing Surveys. 41,3.

Digital Library

[9]

Dairy Australia Publications. 2016. http://www.dairyaustralia.com.au/Industry-information/About-Dairy-Australia/Publications-2.aspx.

[10]

Eurostat: Your key to European statistics. 2016. http://ec.europa.eu/eurostat/about/overview.

[11]

FAO. 2016. http://www.fao.org/statistics/databases/en/.

[12]

Golub et al. 2013. Global climate policy impacts on livestock, land use, livelihoods, and food security. Proceedings of the National Academy of Sciences of the United States of America. 110, 52, 20894--20899.

[13]

International Monetary Fund. 2016. http://www.imf.org/external/index.htm.

[14]

Kerry Group. 2016. http://www.kerrygroup.com/our-company/history/.

[15]

Kepak. 2016. http://www.kepak.com/about-us/.

[16]

Kimball R. and Ross M. 2002. The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling (2nd ed.). Wiley.

Digital Library

[17]

James G., Witten D., Hastie T. and Tibshirani R. 2013. An Introduction to Statistical Learning. Springer.

Digital Library

[18]

Swets J. 1996. Signal Detection Theory and Roc Analysis in Psychology and Diagnostics: Collected Papers. Lawrence Erlbaum Associates.

[19]

Donoghue J., Roantree M., Cullen B., Moyna N., O Sullivan C., and McCarren A. 2015. Anomaly and Event Detection for Unsupervised Athlete Performance Data. Proceedings of LWZ 2015, CEUR Workshop Proceedings. 1458, 205--217.

[20]

Perez J., Berlanga R., Aramburu J. and Pedersen T. 2008. Integrating Data Warehouses with Web Data: A Survey. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING. 20,7.

Digital Library

[21]

Perry B., Graceb D. and Sonesc K. 2013. Current drivers and future directions of global livestock disease dynamics. Proceedings of the National Academy of Sciences of the United States of America. 110,52, 20871--20877.

[22]

Roantree M. and Liu J. 2014. A heuristic approach to selecting views for materialization. Software - Practice and Experience. Vol. 44:10, 1157--1179.

Digital Library

[23]

SeleniumHQ Browser Automation. 2016. http://www.seleniumhq.org/docs/.

[24]

Harrell, Frank E., Jr. 2001. Regression modeling strategies. 9,3. New York: Springer-Verlag.

Digital Library

[25]

United States Department of Agriculture. 2016. http://www.ers.usda.gov/data-products/chart-gallery/detail.aspx?chartId=40037.

Cited By

Bahrpeyma FNgo VRoantree MMcCarren A(2024)A Meta-learner approach to multistep-ahead time series predictionInternational Journal of Data Science and Analytics10.1007/s41060-024-00599-6Online publication date: 9-Jul-2024
https://doi.org/10.1007/s41060-024-00599-6
Wang XWang XWilkes MWang XWang XWilkes M(2020)Enhancing Outlier Detection by Filtering Out Core Points and Border PointsNew Developments in Unsupervised Outlier Detection10.1007/978-981-15-9519-6_7(173-193)Online publication date: 25-Nov-2020
https://doi.org/10.1007/978-981-15-9519-6_7
McCarthy SMcCarren ARoantree M(2019)A Method for Automated Transformation and Validation of Online Datasets2019 IEEE 23rd International Enterprise Distributed Object Computing Conference (EDOC)10.1109/EDOC.2019.00030(183-189)Online publication date: Oct-2019
https://doi.org/10.1109/EDOC.2019.00030
Show More Cited By

Index Terms

Anomaly detection in agri warehouse construction

Recommendations

Alliance Rules for Data Warehouse Cleansing
ICSPS '09: Proceedings of the 2009 International Conference on Signal Processing Systems

Data Cleansing is an activity performed on the data sets of data warehouse to enhance and maintain the quality and consistency of the data. This paper addresses the problems related with dirty data, entrance of dirty data and detection of dirty data in ...
Graph-based anomaly detection
KDD '03: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining

Anomaly detection is an area that has received much attention in recent years. It has a wide variety of applications, including fraud detection and network intrusion detection. A good deal of research has been performed in this area, often using strings ...
Real anomaly detection in telecommunication multidimensional data using data mining techniques
ICCCI'10: Proceedings of the Second international conference on Computational collective intelligence: technologies and applications - Volume PartI

The tremendous amount of data are generated and collected by telecommunication companies. These data include call detail data which describe the calls traversing the telecommunication networks as well as network and customer data which mainly describe ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ACSW '17: Proceedings of the Australasian Computer Science Week Multiconference

January 2017

615 pages

ISBN:9781450347686

DOI:10.1145/3014812

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 January 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ACSW 2017

ACSW 2017: Australasian Computer Science Week 2017

January 30 - February 3, 2017

Geelong, Australia

Acceptance Rates

ACSW '17 Paper Acceptance Rate 78 of 156 submissions, 50%;

Overall Acceptance Rate 204 of 424 submissions, 48%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
156
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bahrpeyma FNgo VRoantree MMcCarren A(2024)A Meta-learner approach to multistep-ahead time series predictionInternational Journal of Data Science and Analytics10.1007/s41060-024-00599-6Online publication date: 9-Jul-2024
https://doi.org/10.1007/s41060-024-00599-6
Wang XWang XWilkes MWang XWang XWilkes M(2020)Enhancing Outlier Detection by Filtering Out Core Points and Border PointsNew Developments in Unsupervised Outlier Detection10.1007/978-981-15-9519-6_7(173-193)Online publication date: 25-Nov-2020
https://doi.org/10.1007/978-981-15-9519-6_7
McCarthy SMcCarren ARoantree M(2019)A Method for Automated Transformation and Validation of Online Datasets2019 IEEE 23rd International Enterprise Distributed Object Computing Conference (EDOC)10.1109/EDOC.2019.00030(183-189)Online publication date: Oct-2019
https://doi.org/10.1109/EDOC.2019.00030
Su SXiao LRuan LGu FLi SWang ZXu R(2019)An Efficient Density-Based Local Outlier Detection Approach for Scattered DataIEEE Access10.1109/ACCESS.2018.28861977(1006-1020)Online publication date: 2019
https://doi.org/10.1109/ACCESS.2018.2886197
McCarthy SMcCarren ARoantree M(2018)Combining Web and Enterprise Data for Lightweight Data Mart ConstructionDatabase and Expert Systems Applications10.1007/978-3-319-98812-2_10(138-146)Online publication date: 9-Aug-2018
https://doi.org/10.1007/978-3-319-98812-2_10
Chaaya GMaalouf H(2017)Anomaly detection on a real-time server using decision trees step by step procedure2017 8th International Conference on Information Technology (ICIT)10.1109/ICITECH.2017.8079989(127-133)Online publication date: May-2017
https://doi.org/10.1109/ICITECH.2017.8079989
Su SXiao LZhang ZGu FRuan LLi SHe ZHuo ZYan BWang HLiu S(2017)N2DLOF: A New Local Density-Based Outlier Detection Approach for Scattered Data2017 IEEE 19th International Conference on High Performance Computing and Communications; IEEE 15th International Conference on Smart City; IEEE 3rd International Conference on Data Science and Systems (HPCC/SmartCity/DSS)10.1109/HPCC-SmartCity-DSS.2017.60(458-465)Online publication date: Dec-2017
https://doi.org/10.1109/HPCC-SmartCity-DSS.2017.60
O’Donoghue JRoantree MMcCarren A(2017)Detecting Feature Interactions in Agricultural Trade Data Using a Deep Neural NetworkBig Data Analytics and Knowledge Discovery10.1007/978-3-319-64283-3_33(449-458)Online publication date: 3-Aug-2017
https://doi.org/10.1007/978-3-319-64283-3_33

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents