Thursday, July 23, 2015
If you need the Dynamics CRM 2015 assemblies you can search for “crm 2015 client” in the Nuget package manager. If you need Dynamics CRM 2013 assemblies – you will need to use the package manager console as outlined in Using Nuget for Dynamics CRM Development Part 1: Nuget basics and useful links
Tuesday, June 16, 2015
When I started playing around with it I was surprised that it seemed to do things quite differently from Power BI for Office 365 since I thought it was simply the next release of the existing Power BI for Office 365 offering. Apparently this is not the case.
Power BI Preview seems to be quite different from Power BI for Office 365 - for a detailed description of differences check out Power BI vs Power BI Preview: what’s the difference – here’s a quick summary:
- Power BI for Office 365 is based on technologies such as Excel and SharePoint and is an integrated part of Office 365, whereas Power BI Preview is built on a separate platform.
- Power BI Preview is using the browser the Power BI Designer as design tool for creating dashboards and reports whereas Power BI for Office 365 mainly relies on Excel as a design tool.
- Power BI Preview also exposes an API which allows you to push data into the Power BI service – for more information check out the Power BI Developer Center. For a good introduction check out Developing for Power BI Overview (Video). This is something which I think is the key enabler for real-time analytics on your data. To stay up to date make sure that you follow the Power BI Development blog
- Power BI Preview has some new data visualizations available such as single number card tiles, combo charts, funnel charts, gauge charts, filled maps and tree maps (Check out Visualization types available in Power BI Reports)
If you check out the official documentation Use Power BI with Microsoft Dynamics Online (Technet) – it seems to focus on the new Power BI Preview but the Microsoft Dynamics CRM templates for Power BI that you can download for free from PinPoint - listed in the second section of the page - seem to be based on Power BI for Office 365. (Use Google Chrome to see the download link – I did not see it when using Internet Explorer 11)
When you actually try to use it in practice together with Dynamics CRM Online you will however encounter some serious limitations which are hopefully resolved by the summer release:
- The Microsoft Dynamics CRM content pack for Power BI preview only exposes a limited set of 10 entities and associated measures – please vote for Dynamics CRM custom field and entity support if you think this should be extended. There exists a workaround where you will export you Dynamics CRM Online data in Excel and then use Excel data in Power BI (See screenshot below for the available entity sets)
- At the moment it is not possible yet to pass in filters to the Power BI Dashboards which seems like an essential requirement for truly embedded analytics in Dynamics CRM using Power BI – you can however vote for this feature on the Microsoft Power BI Support site – Pass filters in URL.
- To make matters even worse, it simply is not possible for the moment to embed Power BI Preview at all into Dynamics CRM – a feature which is available in Power BI for Office 365.
My guess is that the way forward will be Power BI Preview (or name it Power BI 2.0) and it will replace Power BI for Office 365 – you already see it appearing in the license management section of Office 365 (see screenshot below). But for the moment it is still a Preview and no specific release date has been made available so go for Power BI for Office 365 at the moment.
- Connecting to on-premises organizational data from Power BI
- Microsoft Dynamics CRM content pack for Power BI
- Microsoft CRM Online & Power BI Tutorial (Power BI for Office 365)
- Power BI vs Power BI Preview: what’s the difference
- Power BI Workbook Size Limitations
Tuesday, June 09, 2015
Updates and improvements to Dynamics CRM are released twice a year – in what is commonly referred to as the spring and fall release – see Microsoft Dynamics CRM – Roadmap for 2015. Given the new “Cloud first” credo of Microsoft these updates can be a cloud only release as was the case with the Spring 2015 (Carina) release. For Dynamics CRM Online you are required to be on the current version ( n ) or the prior version ( n-1 ) but you have the choice to skip an update – see Manage Dynamics CRM Online Updates. Dynamics CRM on premise follows the standard lifecycle that you are accustomed to (see Microsoft Dynamics Support Lifecycle Policy FAQ and Microsoft Product Lifecycle Search for Dynamics CRM)
To make things a little more interesting the Dynamics CRM product team seems to have chosen to use stars and constellations as code names for the different releases. Code names of the same genre are also used for closely related products to Dynamics CRM such Dynamics Marketing, Social Engagement and Parature Knowledgebase.
Recently Microsoft also changed the naming conventions for their updates and explained the version/build numbers that they are using now and for future releases – check out New naming conventions for Microsoft Dynamics CRM updates. The tables below summarizes the different versions for the moment. As outlined in Greg Olsen his blog post – Microsoft Dynamics CRM 2015 Roadmap – the next version for Dynamics CRM is code named Ara – another interesting tidbit - “Not confirmed by Microsoft, but it is likely that On-Premises installations will have to wait for the CRM ‘ARA’ release during the Fall Wave in order to get the Carina new features and others.”
|Product Name||Version description||Version number||Release or Update||Code Name|
|Microsoft Dynamics CRM Online||Fall ‘13||6.0.0||Major release||Orion|
|Microsoft Dynamics CRM Online||Fall ‘13||6.0.1||Incremental Update||-|
|Microsoft Dynamics CRM Online||Fall ‘13||6.0.2||Incremental Update||-|
|Microsoft Dynamics CRM Online||Spring ‘14||6.1.0||Minor release||Leo|
|Microsoft Dynamics CRM Online||2015 Update (Fall ‘14)||7.0.0||Major release||Vega|
|Microsoft Dynamics CRM Online||2015 Update 1 (Spring ‘15)||7.1.0||Minor release||Carina|
|Microsoft Dynamics CRM Online||t.b.d.||t.b.d.||t.b.d.||Ara|
|Product Name||Version description||Version number||Release or Update||Code Name|
|Microsoft Dynamics CRM (on premise)||2013||6.0.0||Major release||Orion|
|Microsoft Dynamics CRM (on premise)||2013 UR1||6.0.1||Incremental Update||-|
|Microsoft Dynamics CRM (on premise)||2013 UR2||6.0.2||Incremental Update||-|
|Microsoft Dynamics CRM (on premise)||2013 SP1||6.1.0||Minor release||Leo|
|Microsoft Dynamics CRM (on premise)||2015||7.0.0||Major release||Vega|
|Microsoft Dynamics CRM (on premise)||2015 Update 0.1||7.0.1||Minor release||Carina|
|Microsoft Dynamics CRM (on premise)||t.b.d.||t.b.d.||t.b.d.||Ara|
- Dynamics CRM – Get ready for the next release
- Manage Dynamics CRM Online updates
- Customer Driven Updates in CRM Online 2015 Update 1 (Video)
- Microsoft Dynamics CRM 2015, 2013, and 2011 updates: release dates, build numbers and collateral
- Quick tip: determine the version of Microsoft Dynamics CRM
- Dynamics CRM Roadmap – Spring 2015 and beyond
- A few notes on Convergence 2015 Announcements
- Microsoft Dynamics CRM – Roadmap for 2015
Thursday, April 09, 2015
With all the emphasis on Microsoft Power BI – people seem to forget that there still are some other options for setting up a business intelligence solution based on SharePoint available for those of you who can’t go all in for a cloud solution (because of regulations, corporate policies or other reasons). Don’t get me wrong – I do believe that if you are standardized on Microsoft you should follow their “Cloud First” credo. Listed below are a number of links to get you started.
- Configure AdventureWorks for Business Intelligence Solutions with SharePoint 2013
- Create a connection to a data model for Power View
- Power View in SharePoint Server: Create, save and print reports
- Explore the Adventure Works Multidimensional Model by using Power View
- SharePoint Server 2013 BI – Interactive Reports using Power View in Excel 2013
- Install Reporting Services SharePoint Mode for SharePoint 2013
- Supported combinations of SharePoint and Reporting Services Server and Add-in (SQL Server 2012)
- Channel 9 Video – How to create modern BI solutions using Microsoft SharePoint Server 2013, PowerPivot and Power View in Excel 2013
- SharePoint Deep Dive exploration: SharePoint alerting
- SharePoint Deep Dive exploration: looking into the SharePoint userinfo table
SharePoint Server can detect near duplicates of documents and will take this into account when displaying search results. In this post I will delve a little deeper into the underlying techniques being used. An important thing to keep in mind is that the way that duplicate documents are identified has evolved and changed in the different versions of SharePoint.
SharePoint Server 2007 detected duplicates using a commonly used technique called "shingling". This is a generic technique which allows you to identify duplicates or near duplicates of documents (or webpages). Shingling has been widely used in different types of systems and software to identify spams, plagiarism or to enforce copyright protection. A shingle – which is more more commonly referred to as a q-gram – is a contiguous subsequence of tokens taken from a document.
So if you want to see if two documents are similar, you can do this by looking at how many shingles they have in common. You however need to determine how long your subsequence of tokens needs to be – typically a value of 4 is used. This is formalized by using S(d,w), which is the set of distinct shingles of width w which are contained in a document e.g. for the line “a rose is a rose is a rose” – so with w=4, we get the following shingles “a rose is a”, “rose is a rose”, “is a rose is”. If you wan to compare the similarity between two sets, e.g. S(doc1) and S(doc2) which are the sets of distinct shingles of document1 and document2, you can use the Jaccard similarity index (or resemblance index) to define the degree of similarity. A Jaccard index with a value of 0 means that documents are completely dissimilar, whereas 1 points to identical documents. This would however that we would need to calculate the similarity index of each pair of documents – which would be a quite intensive task – to speed up processing a form of hashing is used (for more details take a look at the explanation about near duplicates and shingling)
As items in SharePoint 2007 were indexed, these hashes were stored in the search database. It is not really clear from the documentation whether these hashes only related to the content of an item or to the properties as well (although this blog - Microsoft Office SharePoint Server 2007: Duplicate search results states that it is only on the content of a document). So in SharePoint Server 2007 these hashes were stored in the MSSDuplicateHashes tables.
In SharePoint Server 2013 these hashes are not stored in the MSSDuplicateHashes table anymore but in the DocumentSignature – this is documented in the article Customizing search results in SharePoint 2013. In the next screenshot I have used the and you will notice that although the document title and some metadata are different for the 5 documents, there are only 2 distinct document signatures. This indicates that the shingle is only calculated using the content of documents and not the metadata or the file name (Content By Search web parts don’t seem to use duplicate trimming). The document signature actually contains 4 checksums and if one of the four matches with another document, the document is treated as a duplicate. This also means that when SharePoint search encounters a document for which it is unable to extract the actual contents, it probably is not able to do proper duplicate trimming.
Since SharePoint Server 2013 search result web parts have duplicate trimming activated and SharePoint 2013 is using a quite coarse algorithm for determining a duplicate, you will see some unexpected results. Luckily after installing the SharePoint 2013 Cumulative Update July 2014 you will have the option to de-activate duplicate trimming within the query builder settings.
Another way to accomplish the same thing is by changing the settings for grouping of results. As outlined in Customizing search results in SharePoint 2013, duplicate removal of search results is a part of grouping. So if you specify to group on DocumentSignature, you would be able to show near duplicates (if one of the 4 checksums is different) but still omit the “complete” duplicates.
Thursday, April 02, 2015
Just a quick roundup of some interesting links to articles, whitepapers and videos on Big Data and IoT. I would be amazed if you haven’t heard from Big Data – but still you might still take a look at these introductory blog posts which mainly cover Big Data from a Microsoft perspective.
- Microsoft Big Data – Introducing Windows Azure HDInsight
- Microsoft Big Data – looking into the HDInsight Emulator
- Big Data – getting to the V that really matters
- Microsoft Big Data - Overview of Apache Hadoop components in HDInsight, from Ambari to Zookeeper
Other Big Data and Internet of Things (IOT) links:
- Ten examples of IoT and Big Data working well together – the success or failure of the Internet of Things hinges on big data – says Brian Hopkins, an analyst with Forrester Research
- The internet of things and big data: unlocking the power
- Why big data matters to Boeing and what it means for your next flight
- Our favorite 40+ Big Data use cases – what’s yours?
- How Google is using big data and machine learning to aid drug discovery
- Whitepaper: Process automation and IoT – Yokogawa’s approach
- How Pfizer is using Big Data to Power Patient Care
- Platform ecosystems will be the revolutionary foundation for IoT (Accenture Technology Vision 2015)
- Making the Internet of Things – TweetHeart – A NeoPixel Heart that is twitter sensitive
- Leveraging the Internet of Everything to create a better customer experience
- Lab of Things enables research and teaching
- Bosch pools Industry 4.0 expertise in the “Connected Industry” innovation cluster
- Data Science for IoT: the role of hardware in analytics
- The analytics of things (Deloitte)
- The Internet of Things: Cities as a Platform (Sogeti labs)
- Making the Internet of Things (Part 1) – Exploring the littoral space
- Internet of Things or Things on the Internet?
- IT Tomorrow – Internet of Things (Video in Dutch)
- 6 predictions for the $125 billion Big Data analytics market in 2015
Tuesday, March 31, 2015
- Ambari – provides provisioning, monitoring and management layer on top of Apache Hadoop clusters. It provides a web interface for easy management as well as a REST API.
- Flume – allows you to collect, aggregate and move large volumes of streaming data into HDFS in a fault tolerant fashion.
- HBase – provides NoSQL database functionality on top of HDFS. It is a columnar store, which provides fast access to large quantities of data. HBase tables can have billions of rows and these rows can have almost unlimited number of columns.
- HCatalog – provides a tabular abstraction on top of HDFS. Pig, Hive and Mapreduce use this layer to make it easier to work with files in Hadoop. HCatalog has been merged into the Hive project. Hive uses it kind of a like a master database. For more details check out Apache HCatalog – a table management layer that exposes Hive metadata to other Hadoop applications.
- Hive – allows you to perform data warehouse operations using HiveQL. HiveQL is a SQL like language and provides an abstraction layer on top of MapReduce. Hive allows you to use Hive tables to project a schema onto the data (schema on read). Through the use of HiveQL you can view your data as a table and create queries just as you would in a normal database with support for selects, filters, group by, equi-joins, etc…. Hive inherits schema and location information from HCatalog. Hive will act as a bridge to many BI products which expect tabular data. One of the recent developments around Hive is the Stinger initiative – its main aim is to deliver performance improvements while keeping SQL compatibility
- Kafka – is a fast, scalable, durable and fault-tolerant messaging system. It is commonly used together with Storm and HBase for stream processing, website activity tracking, metrics collection and monitoring or log aggregation. It is provides similar functionality as AMQP, JMS or Azure Event Hub
- Mahout – the goal of Mahout is build scalable machine learning libraries. The main machine learning use cases Apache Mahout support are recommender systems (people who buy x also buy y), classification (assigning data to discrete categories e.g. is a credit card transaction fraudelent or not) and clustering (grouping unstructured data without any training data). For more details take a look at Introducing Mahout (IBM)
- Oozie – enables you to create repeatable, dynamic workflows for tasks to be performed in a Hadoop cluster. An Oozie workflow can include Sqoop transfers, Hive jobs, HDFS commands, Mapreduce jobs, etc … Oozie will submit the jobs but Mapreduce will execute them. Oozie also has built-in callback and pollback mechanisms to check for the status of jobs
- Pegasus provides large scale graph mining capabilities by offering important graph mining algorithms such as degree calculation, pagerank calculation, random walk with restart (RWR), etc .. Most graph mining algorithms have limited scalability, they support up to millions of nodes. Pegasus billion-node graphs. Graphs (also referred to as networks) are everywhere in real life going from web pages, social networks, biological networks and many more… Finding patterns, rules etc within these networks allow you to rank web pages (or documents), measure viral marketing, discover disease patterns, etc … The details of Pegasus can be found in the white paper Pegasus: a peta-scale graph mining system – implementation and observations.
- Pig is developed to make data analysis on Hadoop easier. It is made up of two components: a high level scripting language (which is called Pig Latin but most people just reference it as Pig) and an execution environment. Pig Latin is a procedural language which allows you to build data flows, it contains a number of built in User Defined Functions (UDFs) to manipulate data. These UDFs allow you to ingest data from files, streams or other sources, make selections and transform the data. Finally Pig will store the results back into HDFS. Pig scripts are translated into a series of MapReduce jobs that are run on Apache Hadoop. Users can create their own functions or invoke code in other languages such as JRuby, Jython and Java. Pig will gives you more control and optimization over the flow of the data than Hive does.
- RHadoop – is a collection of R packages that allow users to manage and analyze data with Hadoop in R, including the creation of map-reduce jobs. Check out Step-by-step guide to setting up an R-Hadoop system and Using RHadoop to predict website visitors to get started with some hands-on examples.
- Storm – distributed real-time computation system, it supports a set of common stream analytics operations, provides guaranteed message processing with support for transactions. It was originally created by Nathan Marz (see History of Apache Storm and lessons learned) – the guy who cam up with the term Lambda architecture for a generic, scalable and fault tolerant data processing architecture.
- SQOOP – was built to transfer data from relational structured data stores (such as SQL Server, MySQL or Oracle) to Apache Hadoop and vice versa. Because Sqoop can handle database metadata, it is able to perform type-safe data movement using the data types specified in the metadata.
- Zookeeper – manages and store configuration information. It is responsible for managing and mediating conflicting updates across your Hadoop cluster.
Thursday, March 26, 2015
Whereas marketing and sales as well as financial departments have been using advanced analytics for quite a while, it seems that HR is still in one of the early maturity phases of analytics usage. This is a view which seemed to be shared by CEOs. In a recent study CEOs gave their HR department a 5.9 (out of 10) for their analytical skills. (See CEO niet overtuigd van analytische skills HR )
Whereas HR controls a lot of data (and needs to keep it up to date) it does not seem to be able to use this data to provide strategic advise to the board of directors. HR can only deliver truly added value by providing data-driven insights regarding people that are both compelling to business leaders and actionable by HR. This is a view which is also quite nicely outlined by consultancy firm Inostix in their HR Analytics Value Pyramid (See The HR Analytics Value Pyramid (Part 3) ). To make sure that HR team stays current and viable, they will need to adopt a whole need set of skills of which analytics is just one (See The reskilled HR team – transform HR professionals into skilled business consultants and the capability gap across the 2015 Human Capital Trends)
In a number of upcoming posts I will delve a little deeper into this topic and will show some practical examples of how you can realize some quick wins without a huge upfront investment.
- What we learned about HR Analytics in 2014
- 17 differences between HR Metrics and Predictive HR Analytics
- Datafication of human capital
- Top 72 HR Analytics Influencers Part 3
- Business need to make better use of analytics to predict what they need than just recruiting
- Sink or swim: a tidal wave of technology is shaping HR
- How important is data analytics to the future of HR?
- Six takeaways from the HR Analytics Innovation Summit
- Is HR ready for the big data and analytics revolution?
- Making the business case for predictive talent analytics
- Leveraging predictive analytics to avoid a major point of hiring failure
SharePoint Saturday 2015 : How to build your own Delve, combining machine learning, big data and SharePoint
BIWUG is organizing the fifth edition of SharePoint Saturday Belgium – this year in Antwerp – for more information check out the site http://www.spsevents.org/city/Antwerp/Antwerp2015/ . Here is the excerpt of the session I will be delivering.
How to build your own Delve: combining machine learning, big data and SharePoint
You are experiencing the benefits of machine learning everyday through product recommendations on Amazon & Bol.com, credit card fraud prevention, etc… So how can we leverage machine learning together with SharePoint and Yammer. We will first look into the fundamentals of machine learning and big data solutions and next we will explore how we can combine tools such as Windows Azure HDInsight, R, Azure Machine Learning to extend and support collaboration and content management scenarios within your organization.
- Microsoft Azure Machine Learning – the power to predict
- Data science dojo – Beginning AzureML video series
- Big Data – Beyond the hype, getting to the V that really matters
- Microsoft Big Data – Introducing Windows Azure HDInsight